# Comparison of Machine Learning's- and Humans'- Ability to Consistently Classify Anomalies in Cylinder Locks

Tim Andersson, Markus Bohlin, Tomas Olsson, Mats Ahlskog

<figure><img src="image_1.png" /></figure>

## To cite this version:

Tim Andersson, Markus Bohlin, Tomas Olsson, Mats Ahlskog. Comparison of Machine Learning's- and Humans'- Ability to Consistently Classify Anomalies in Cylinder Locks. IFIP International Conference on Advances in Production Management Systems (APMS), Sep 2022, Gyeongju, South Korea. pp.27-34, 10.1007/978-3-031-16407-1_4. hal-05386012

HAL Id: hal-05386012

https://inria.hal.science/hal-05386012v1

Submitted on 27 Nov 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Comparison of Machine Learning's-
and Humans' - Ability to Consistently Classify
Anomalies in Cylinder Locks

Tim Andersson(✉), Markus Bohlin, Tomas Olsson, and Mats Ahlskog

Mälardalen University, Eskilstuna, Sweden

{Tim.Andersson, Markus.Bohlin, Tomas.Olsson, Mats.Ahlskog}@mdu.se

**Abstract.** Historically, cylinder locks' quality has been tested manually by human operators after full assembly. The frequency and the characteristics of the testing procedure for these locks wear the operators' wrists and lead to varying results of the quality control. The consistency in the quality control is an important factor for the expected lifetime of the locks which is why the industry seeks an automated solution. This study evaluates how consistently the operators can classify a collection of locks, using their tactile sense, compared to a more objective approach, using torque measurements and Machine Learning (ML). These locks were deliberately chosen because they are prone to get inconsistent classifications, which means that there is no ground truth of how to classify them. The ML algorithms were therefore evaluated with two different labeling approaches, one based on the results from the operators, using their tactile sense to classify into 'working' or 'faulty' locks, and a second approach by letting an unsupervised learner create two clusters of the data which were then labeled by an expert using visual inspection of the torque diagrams. The results show that an ML-solution, trained with the second approach, can classify mechanical anomalies, based on torque data, more consistently compared to operators, using their tactile sense. These findings are a crucial milestone for the further development of a fully automated test procedure that has the potential to increase the reliability of the quality control and remove an injury-prone task from the operators.

**Keywords:** Machine learning · Binary classification · Torque data · Multiple experts · Cylinder lock

## 1 Introduction

The result from the production process for mechanical parts is dependent on the quality of the material, machinery, personnel, environment, and measurements. These factors will eventually cause dimensional deviations that exceed the given tolerances [1]. A product that is an assembly of several parts will inherently be sensitive to these deviations. In this article, a type of pin-tumbler cylinder lock mechanism is considered [2]. A lock consists of tens of moving parts, see Fig. 1, where a deviation from the given tolerances
in the production process can result in increased friction or an unsmooth jerky motion when the lock is operated. The locks get tested at the very end of the assembly process, before packaging for delivery, to ensure the quality of the product. It is not possible to detect faults by a visual inspection, instead, testing is done by experienced operators that subjectively decide upon the quality of a lock by using their tactile sense when turning the lock mechanism with the corresponding key, an operation which unfortunately can lead to substantial wear of the operators' wrists. The faults that can occur during the production of the lock mechanism can easily be detected when the lock needs relatively high torque to operate, but in other cases, it is not easy to detect faults, such as when the lock has not been oiled properly. Therefore, different operators might classify these locks as both working and faulty. The result of inconsistent classifications can lead to greater variation in the expected lifetime of a lock. The purpose of this research is to investigate the possibility of an automated solution to objectively test locks by analyzing the required torque to turn the key 360 degrees. An automated method can potentially increase the reliability in the testing phase and remove a monotonous and injury-prone task from the operators. When analyzing sensor data, it is common to use Machine Learning (ML) because of its ability to effectively detect patterns in high dimensional datasets [3]. The main research questions this study seeks to answer are, how consistent can the operators classify locks, using their tactile sense, compared to an automated solution, using torque data and ML, and is the operator's tactile sense suitable to use as a source of labels for the training of the ML algorithms? This study evaluates three different ML algorithms' ability to consistently classify mechanical anomalies in the locks with two different approaches to create labels for the training dataset, the results are then compared to the operators' ability. The company where this study was done, is one of the largest manufacturers in the world of different types of locks. Furthermore, to the best of our knowledge, this paper is the first published study on an automated solution for quality control of locks. It is therefore expected that the results from this study are of great interest for this industry but also, for other applications where fault detection is done by rotating a mechanical system.

<figure><img src="image_4.png" /><figcaption>Fig. 1. Illustration of the cylinder lock that was used in this study [4].</figcaption></figure>
## 2 Literature Review

Today's harsh global competition forces the industry to constantly improve its cost-efficiency. To achieve this, it is important to minimize downtime of the systems and machinery which has led to constant monitoring of products' health status to predict when corrective actions are needed based on sensor data, this is called Predictive Maintenance (PdM) [3]. This is similar to what this study seeks to achieve in the context of locks. Although the PdM method can lead to an improved lifetime and downtime its success is highly dependent on the sensor data and the gained insights the model is based upon [5]. The PdM method often needs to handle high-dimensional datasets and ML has proven to be successful for such tasks which are why ML algorithms are often used [3]. Two common methods in ML are called supervised- and unsupervised- learning [6]. In this study, both a supervised and an unsupervised learning approach were used. Supervised learning is an agent that learns a function based on a set of input-output pairs, i.e. labeled observations, that can map the input to output. These input-output pairs are assumed to be correct and can therefore act as a ground truth. However, if the data are labeled by multiple experts, as in this study, the labels can be contradictory and affect the classification results negatively. To counteract this problem there exist different methods to develop consensus models, such as majority voting, which was used in this study, but also more complex approaches [7]. In contrast, an unsupervised learner finds patterns in the input without any explicit feedback. Unsupervised learners often have the task of clustering observations based on similar properties. In [3] the authors reviewed 36 different papers that have used ML algorithms in PdM implementations where it can be seen that Support Vector Machine (SVM), Decision Tree (DT), K-Nearest Neighbor (KNN) and K-Means have been frequently implemented. When developing ML algorithms for classification tasks every class should be represented equally often in the dataset to avoid the algorithm getting biased towards a specific class, and hence poor performance [8]. In essence, it is in practice often easier to get data regarding a normal condition of a system compared to an abnormal condition, but in many cases, it is the anomalies that are of interest. To solve skewed datasets a synthetic dataset can be generated based on the original dataset with methods for balancing minority classes, for example, SMOTE [9], which is the approach used in this paper.

## 3 Method and Implementation

To evaluate the operators' ability to consistently classify mechanical anomalies, using their tactile sense, a test was designed where 9 different operators were instructed to classify the same collection of 46 unused locks as either 'faulty' or 'working' according to their opinion of the quality. The set of locks was deliberately chosen because they had not been lubricated properly for three years after they were stored, which should result in a higher frequency of minor mechanical anomalies that can typically be difficult to detect with the human tactile sense. The test resulted in 413 votes in total (one of the operators forgot to classify a single lock). The distribution of all votes between the two classes were 84% 'working' and 16% 'faulty'. For assessing the inconsistency of the operators, we compute the majority vote fraction, that is, the percentage of votes consistent with
the majority vote for each lock. A low majority vote fraction indicates a low consistency between operators and vice versa. For instance, if 5 voted ‘working’ and 4 voted ‘faulty’, then the majority vote is ‘working’ and the majority vote fraction is about 56%, which is a low consistency. Given that we have only 9 operators, this is the lowest possible value of consistency. In contrast, if all operators voted the same, we would have a fraction of 100% which is the highest possible value of consistency.

## 3.1 Collecting, Processing and Labeling of Torque Data

The operators use their tactile sense to detect anomalies related to the change in the needed torque to turn the key when the lock is faulty, which is the reason why the automated solution used a torque measuring machine to collect data. To get a similar evaluation of the automated solution’s consistency, we collected 9 different torque measurements per lock that then could be classified by the ML algorithm in order to get 9 different “votes” corresponding to the votes from the 9 operators (however, due to the missing data, one of the locks was tested 8 times). The machine collects the data with a torque transducer by holding the cylinder in a fixed position while turning the cylinder core with the corresponding key during a constant angular velocity of 30 degrees per second for 360 degree-turn in an angular position system with a resolution of 0.1 degrees resulting in 3600 measuring points for each lock. All measuring instruments are subjected to a specific degree of variations in their measuring results depending on the instrument’s precision and surrounding noise. This means that 9 repeated torque measurements from the same lock will all be unique. The torque transducer used in this study has a range of 0.1–200 Nm and a sensitivity of 0.02 Nm [10]. Approximately 0.5% of the torque measurements for different angles lost data when they experienced a sudden change in friction resulting from the lack of lubrication. This is due to the cycle time of 1 ms that the computer reads the torque transducer with. The spring-effect that occurs from the sudden change of friction will force the lock to rotate too fast and therefore resulting in missed readings. This was solved by using linear interpolation [11] with the neighboring values. It is a common understanding that an ML algorithm will not perform any better than the labeled data it has been trained on. Since it is already known that the subjective evaluations of the locks’ quality done by the operators’ tactile sense can give inconsistent results, the ML algorithms are trained using two different labeling methods. This approach makes it possible to evaluate how the two different labeling methods will affect the classification results from the ML algorithms.

**Tactile Sense Labeling.** The operators have only knowledge of classifying the locks based on their tactile sense, so an alternative method was used to label the torque measurements. The first set of reference labels is created based on the classification results from the operators. The torque measurements from a specific lock are labeled such that it has the same fraction between the two different labels as the operators’ labels for the same lock. To minimize the effect on the training result from variations in the measurements that originates from a specific lock, the torque measurements were randomly chosen to belong to a specific class fraction. For example, lock x was labeled ‘faulty’ 5 times and 4 times as ‘working’ by the operators when they used their tactile sense, this means that 5 torque measurements were randomly labeled as ‘faulty and 4 were
randomly labeled as ‘working’ for the same lock. This approach was repeated 10 times to be able to calculate an average result from 10 different evaluations for each of the sets of labels for the torque dataset.

**Cluster Labeling.** The second set of reference labels was created by letting an unsupervised learner, named k-means, divide the dataset into two different clusters such that the sum of the squared Euclidian distances from each measurement-point to the cluster centroid is minimized. By visual inspection of the torque diagrams from the two clusters, an expert identified one cluster that contained torque measurements with more torque anomalies, see Fig. 2. This cluster’s torque measurements were labeled as ‘faulty’ and the other cluster as ‘working’. This resulted in a dataset with 37% of the torque measurements labeled as ‘working’ and 63% labeled as ‘faulty’. The method of analyzing torque diagrams from locks has not been evaluated before which means there is no established ground truth so the size and labels for the clusters are not important, the main focus is consistency.

<figure><img src="image_5.png" /><figcaption>**Fig. 2.** Torque diagrams labeled as ‘working’ (left) and ‘faulty’ (right) by an expert after clustering using the k-means method.</figcaption></figure>

## 3.2 Training and Evaluation

The ML algorithms SVM, DT and KNN were chosen in this study because they are commonly used in similar applications. The ML algorithms were initially trained on a synthetic dataset based on the original torque dataset using a systematic parameter variation with a Bayesian Optimization algorithm [12] to get suitable parameters for the ML algorithms. The purpose of this was to give the different ML algorithms optimal circumstances for the training on the real dataset to be able to compare the methods’ results knowing that they probably have the optimal settings. All the ML algorithms were tested by excluding all the data gathered from one lock from the dataset and the rest of the data from all the other locks were used as a training dataset i.e. leave-one-out cross-validation [13]. The training dataset was balanced with the method SMOTE. This process was done iteratively for every lock in the dataset until all the torque measurements from each lock have been classified. The final labels from an ML algorithm for all 46 locks
were chosen based on how the majority of the torque measurements that originate from the same lock were classified. Similar to the operators, the uncertainty and inconsistency of the ML algorithms were assessed using the majority label fraction for each lock i.e. consistency, and then the average consistency for the two classes was calculated. The processing of the collected data and the implementation of the algorithms were all done in MATLAB [14].

## 4 Results

In the tables below are the experimental results from the classification of locks. The tables are structured such that the results related to a dataset created with a specific labeling approach are located in the corresponding column named after the used labeling approach. Table 1 shows how consistent the torque measurements that originate from the same lock were labeled at mean consistency. The results show that the tactile sense labeling approach resulted in a dataset with a low mean consistency of 86 and 61% for the ‘working’ and ‘faulty’ classes respectively compared to the corresponding values for the cluster labeling approach which are 97 and 98%. It can also be seen in Table 1 that the three ML algorithms SVM, DT and KNN achieved higher mean consistency when they were trained with the dataset from the cluster labeling approach.

Table 2 shows the results from the classification of locks in terms of classification loss. The classification loss is expressed as the percentage of locks that were not classified correctly with respect to the reference labels created from the two labeling approaches. The results in Table 2 show that the ML algorithms suffered high classification loss and great differences in their results when they were trained with the dataset from the tactile sense labeling approach. From the results in the tables, it can be seen that when the ML algorithms SVM, DT and KNN were trained with the dataset created by the cluster labeling approach they all performed similarly to each other in terms of consistency but SVM outperformed DT and KNN in terms of classification loss.

<table><caption>**Table 1.** Mean (μ) and standard deviation of consistency for each ML algorithm</caption><thead><tr><th rowspan="3"></th><th colspan="4">Tactile sense labeling [%]</th><th colspan="4">Cluster labeling [%]</th></tr><tr><th colspan="2">Working</th><th colspan="2">Faulty</th><th colspan="2">Working</th><th colspan="2">Faulty</th></tr><tr><th>μ</th><th>σ</th><th>μ</th><th>σ</th><th>μ</th><th>σ</th><th>μ</th><th>σ</th></tr></thead><tbody><tr><th>Dataset</th><td>86</td><td>11</td><td>61</td><td>8</td><td>97</td><td>7</td><td>98</td><td>8</td></tr><tr><th>SVM</th><td>100</td><td>3</td><td>—</td><td>—</td><td>95</td><td>11</td><td>98</td><td>8</td></tr><tr><th>KNN</th><td>80</td><td>16</td><td>81</td><td>18</td><td>94</td><td>14</td><td>95</td><td>13</td></tr><tr><th>DT</th><td>86</td><td>15</td><td>75</td><td>18</td><td>94</td><td>12</td><td>95</td><td>13</td></tr></tbody></table>
<table><caption>**Table 2.** Classification loss i.e. percentage of locks that were not classified correctly for each class.</caption><thead><tr><th rowspan="2"></th><th colspan="2">Tactile sense labeling [%]</th><th colspan="2">Cluster labeling [%]</th></tr><tr><th>Working</th><th>Faulty</th><th>Working</th><th>Faulty</th></tr></thead><tbody><tr><th>SVM</th><td>0</td><td>100</td><td>0</td><td>0</td></tr><tr><th>KNN</th><td>45</td><td>50</td><td>17</td><td>3</td></tr><tr><th>DT</th><td>20</td><td>100</td><td>12</td><td>10</td></tr></tbody></table>

# 5 Conclusion and Discussion

In this study, a comparison has been made regarding how consistent the operators can classify mechanical anomalies in locks into two groups, named ‘faulty’ and ‘working’, using their tactile sense compared to an automated solution using torque measurements and machine learning. The results from this study show that operators are likely to achieve about 74% consistency on average when asked to repeatedly classify mechanical anomalies in locks resulting from the lack of lubrication. The corresponding values for the automated solution are 96% when trained using the cluster labeling approach and 88% when trained with the tactile sense labeling approach. However, when ML algorithms were trained with the tactile sense labeling approach, they suffered 53% classification loss, on average, as compared to only 7% with the cluster labeling approach. An observation during the experiments that could partially explain this was that the operators rotated the locks faster and with greater variation in their techniques compared to the automated test machine. When the machine rotated the locks, some of them started to oscillate, this did not happen for the operators. It was possible to recreate this phenomenon when an operator rotated the key inside the lock with the same angular velocity as the test machine. This shows that the angular velocity is vital for certain behaviors to arise and can be an explanation for why the operators are not as consistent as the test machine to classify the locks since they all have different testing techniques. From the results in this paper, it can be concluded that the automated solution can measure torque variations in the locks such that the ML algorithms (SVM, DT and KNN) can achieve higher consistency in their classifications, compared to the operators. Furthermore, it seems promising to use a more objective labeling approach such as the cluster labeling approach augmented with expert judgment. An established ground truth regarding how to evaluate torque diagrams could be used in a similar approach to categorize different types of mechanical anomalies automatically, and, possibly, with even higher consistency and precision than either of the variants hitherto investigated. The approach of using torque measurements and ML to detect anomalies of a rotating mechanical system has been done before [15] but, mapping each measurement to an angular position seems to be less common. The advantage of this mapping, compared to using regular time series data, is that there are no phase shifts between the observations which minimize the need for preprocessing of the data.
## 5.1 Threat to the Validity and Future Work

This study is done on a small sample of 46 locks and with just 9 operators to compare with. These factors are an obvious threat to the external validity of this study. The test was performed by letting the operators follow a set of instructions without supervision. This means that there is no way of knowing if they followed the instructions correctly or if they performed the test with their best intentions and during the training, synthetic data were used to get balanced datasets, these factors are a threat to the internal validity of the study. The next step in this research is to gather more data from locks directly from the assembly line where the faults are likely to be more distinct and therefore consistently classified by the operators, but also further research needs to be made on how to recognize a working/faulty lock based on torque data. This would make it possible for the operators to use the machine to evaluate the quality by visual inspection of the torque diagram instead of using their tactile sense. This removes the monotonous task of manual testing of these locks during the development of the ML algorithms but also provides the possibility for a more reliable dataset for minor mechanical anomalies which were not consistently classified with the current tactile sense testing method.

## References

1. Krynke, M., Knop, K., Mielczarek, K.: Identifying variables that influence manufacturing product quality. Prod. Eng. Arch. **4**, 22–25 (2014)

2. Pulford, G.W.: High-Security Mechanical Locks: An Encyclopedic Reference. Elsevier Inc., San Diego (2007)

3. Carvalho, T.P., et al.: A systematic literature review of machine learning methods applied to predictive maintenance. Comput. Ind. Eng. **137** (2019). https://doi.org/10.1016/j.cie.2019.106024

4. Case company: Service manual. Case company (2016)

5. Wickern, V.M.Z.: Challenges and reliability of predictive maintenance. Rhine-Waal (2019)

6. Russell, S., Norvig, P.: Forms of learning. In: Artificial Intelligence a Modern Approach, 3rd edn, pp. 693–695. Prentice Hall, New Jersey (2010)

7. Valizadegan, H., Nguyen, Q., Hauskrecht, M.: Learning classification models from multiple experts. J. Biomed. Inf. **46**, 1125–1135 (2013). https://doi.org/10.1016/j.jbi.2013.08.007

8. Japkowicz, N., Stephen, S.: The class imbalance problem: a systematic study. Intell. Data Anal. **6**, 429–449 (2002)

9. Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res. **16** (2002). https://doi.org/10.1613/jair.953

10. Messtechnik, H.B.: T21WN Torque Transducer. https://www.hbm.com/en/7343/t21wn-torque-meter-with-cylindrical-shaft-stubs, Accessed 29 Apr 2022

11. Caruso, C., Quarta, F.: Interpolation methods comparison. Comput. Math. Appl. **35**, 109–126 (1998). https://doi.org/10.1016/S0898-1221(98)00101-1

12. MathWorks: Bayesian Optimization Algorithm. https://se.mathworks.com/help/stats/bayesian-optimization-algorithm.html#bvaz8tr-1, Accessed 29 Apr 2022

13. Russel, S., Norvig, P.: Leave-one-out cross-validation. In: Artificial Intelligence a Modern Approach, 3rd edn, p. 708. Prentice Hall, New Jersey (2010). Accessed 29 Apr 2022

14. MathWorks: MATLAB. https://se.mathworks.com/products/matlab.html, Accessed 29 Apr 2022

15. Reddy, M.C.S., Sekhar, A.S.: Detection and monitoring of coupling misalignment in rotors using torque measurements. Measurement **61**, 111–122 (2015). https://doi.org/10.1016/J.MEASUREMENT.2014.10.031