# On the Track to Application Architectures in Public Transport Service Companies
## 4 Current State of Use-Cases from Different Transportation Service Providers
### AI in Rail-Inspect at SBB
#### 4.3.1 Data and Model Life-Cycle  
Data is most critical for the success of ML-models and needs to be proactively managed during its entire life-cycle, from its acquisition to its incorporation as part of the model development. Figure 7 shows a section of the SDLC of Figure 2, which illustrates the image life cycle. It is not only the quantity, but also the quality of data which needs to be managed and the transition from big data to good data is crucial [22]. There are two main issues related to data quality: (1) non-consolidated definition of objects of interest and (2) inconsistent labelling. Usually data is labelled by different experts. Without a systematic business consolidation we observed quite often very heterogeneous definitions of the objects of interest. As depicted in Figure 8, two experts can have a different view about the size of the defects to categorize an object as being a defect or not. Such different views consequently not only lead to low quality models but also to different opinions from experts, i.e., some will complain that not all defects were detected (false-negatives) and some will complain that too many defects were detected (false-positives). The consolidation is a crucial prerequisite for successful models. The consistency of the labels has also a great impact on the model performance. Quite often we observed many inconsistent labelling, which include too large or to narrow bounding boxes as well as labelling the same defect more than once (Figure 9). Figure 6: Difficulty in detecting defects based on image data with AISI. The experts need clear guidelines on how to label the data and must be aware of the consequences of inconsistent labelling on model performance. Moreover, the tedious labelling is boring for humans and concentration quickly drops off [10]. Assessing the quality of data and model performance are not one-shot activities, but need to be continuously monitored and managed during their entire life-cycle. As measurement data are inferred with the deployed models, their performance needs to be assessed and based on these results, additional data might be necessary (Figure 7). After that, the models need to be retrained in order to close the gaps identified by the assessments of the model performance. The following reasons lead to the necessity of this process. First, there might be a data and concept shift [23; 24] causing different data distributions as well as different mappings of data to labels over time. In our case, if we change the cameras, the resolution of images will most probably increase and consequently trigger a change to consider even smaller defects with a length of 1 mm (see Figure 8) to be reported. In the first case, the data distribution has changed, whereas in the second case the mapping of data to labels is different. Furthermore, in many cases, existing data does not cover the entire target application domain or problem space. In AISI, we initially only covered a subset of the entire network. As we rolled-out the model to more tracks we faced new defect situations, which were not covered by the initial model and caused a performance degradation of the  
Figure 8: A non-consolidated definition of objects of interest. Figure 7: Model and Data Life-Cycle in the SDLC for TR19 solutions. Figure 9: Inconsistent labelling—inconsistent size of boxes confusing the ML model. model on these extended network tracks. As depicted in Figure 2, such boundary events caused by drifts in data or changing rules can trigger additional investigations in the ML as well as KE sub-processes to constantly adapt to the changing business requirements. This is the reason why we introduced two complementary procedures, which we call crowd-evaluation and expert-labelling. Both procedures make the data a first-class citizen [25] and support the aforementioned continuous management process. The goal of crowd-evaluation is to let different experts label data based on the model performance in a crowd-manner (assign the task to more than one expert).