DOI: 10.5281/zenodo.8238653
title: Mudestreda Multimodal Device State Recognition Dataset

description: Mudestreda Multimodal Device State Recognition Dataset 



obtained from real industrial milling device with Time Series and Image Data for Classification, Regression, Anomaly Detection, Remaining Useful Life (RUL) estimation, Signal Drift measurement, Zero Shot Flank Took Wear, and Feature Engineering purposes.


The official dataset used in the paper "Multimodal Isotropic Neural Architecture with Patch Embedding" ICONIP23.  

Official repository: https://github.com/hubtru/Minape  

Conference paper: https://link.springer.com/chapter/10.1007/978-981-99-8079-6_14  

 

Mudestreda (MD) | Size 512 Samples (Instances, Observations)| Modalities 4 | Classes 3 |  

 

Future research: Regression, Remaining Useful Life (RUL) estimation, Signal Drift detection, Anomaly Detection, Multivariate Time Series Prediction, and Feature Engineering.


 

Notice: Tables and images do not render properly. 

Recommended: `README.md` includes the Mudestreda description and images `Mudestreda.png`  and `Mudestreda_Stage.png`.

 



Data Overview



Task: Uni/Multi-Modal Classification

Domain: Industrial Flank Tool Wear of the Milling Machine

Input (sample): 4 Images: 1 Tool Image, 3 Spectrograms (X, Y, Z axis)

Output: Machine state classes: `Sharp`, `Used`, `Dulled`

Evaluation: Accuracies, Precision, Recal, F1-score, ROC curve



Each tool's wear is categorized sequentially: Sharp → Used → Dulled.




The dataset includes measurements from ten tools: T1 to T10.


Data splitting options include random or chronological distribution, without shuffling.

Options:



   Original data or Augmented data

   Random distribution or Tool Distribution ([see Dataset Splitting](#dataset-spliting))




 
description: Abstract

The rapid advancement in multimodal datasets primarily featuring images, audio, and text, coupled with transformer-based architectures, has been undeniable. However, many real-world industrial scenarios necessitate a distinct approach, focusing on industrial-grade imagery and multivariate time series data from sensors, suitable for deployment on efficient edge devices. This shift is driven by two major limitations of existing multimodal open-source datasets:

(i) they are often constrained to specific modalities, typically centered around video data, limiting their adaptability,

and (ii) they predominantly feature general-purpose objects, which do not address domain-specific and industrial challenges effectively.


Addressing these gaps, we introduce the Mudestreda dataset, a novel multimodal industrial device state recognition resource. Mudestreda uniquely showcases the multimodal nature of industrial tool wear, comprising 512 samples with four-dimensional observations: three force signal sequences and one RGB image of a shaft milling tool, collected over five weeks from a Production Centrum.


 

Our dataset reveals a strong correlation between tool wear and force amplitude increases, signifying progressive tool deterioration. Mudestreda models tool wear across three states: sharp, used, and dulled. Each tool's transition is captured as a temporal sequence, enabling two distinct approaches: treating each observation independently or as correlated sequential data. The former facilitates learning individual sample representations (tool image and three spectrograms), while the latter allows for modelling state dependencies, ideal for memory-based or recurrent neural network architectures.


Additionally, Mudestreda supports experimentation with multimodal classifiers, enabling independent unimodal analysis and comparative performance assessment across varying numbers of modalities. In our current work, we emphasize the dataset's capability in multimodal classification, evaluating the impact of modality count on performance.
description: Future Work



We aim to extend our research to Remaining Useful Life (RUL) estimation, considering the final observation of each tool (T1-T10) as the endpoint of its lifecycle. We also plan to explore fault and anomaly detection, treating the 'Dulled' class as an anomaly.


Our dataset design, with tools T1-T8 for training, T9 for validation, and T10 for testing, simulates zero-shot scenarios, challenging models to generalize to new, unseen conditions. This setup is vital for studying zero-shot learning and its industrial applications, like predictive tool wear adaptation.


Future endeavours will also focus on feature engineering, developing a Diagnostic Feature Designer app for feature importance assessment and selection. This includes extracting blade shapes from RGB tool images, segmenting flank wear, and processing time series data through various techniques like temporal window-based features, Fourier Transforms, Wavelet Transforms, Decomposition, Domain-Specific Features (e.g. RMS (Root Mean Square) Forces, Harmonic Analysis of signals, Spectral Kurtosis, Enveloping or Demodulation Techniques and Cross-Correlation Features.

 


We are also expanding the dataset to include additional modalities such as:



Acoustic emission to monitor the sound emitted by the machinery,

Temperature Monitoring as high temperatures can indicate excessive friction,

Power consumption that can be used to indicate the machine's efficiency,

Torque and Load Analysis can reveal issues with the load on the machinery,

Speed Variations may signal problems with tool wear,

Contamination Levels of machining residues.



We are working on the multi-regression labels:



Flank wear [µm],

Gaps [µm],

Overhang [µm],  


These additions aim to enrich the dataset for multivariate time series prediction, signal drift measurements, and precise flank tool wear estimation.


In conclusion, the Mudestreda dataset presents a unique industrial multimodal device state recognition benchmark for advancing machine learning, neural network and feature engineering methodologies. It is designed for developing and testing lightweight, fast, and explainable algorithms suitable for practical, domain-specific industrial applications and real-life edge-device implementation.
description: Use Cases 

 

 






Input
Model
Output






4 Images (1 Tool Image, 3 Spectrograms (X, Y, Z))






Classification Model






Class (Flank Tool Wear: `Sharp`, `Used`, `Dulled`)








3 Spectrograms (X,Y,Z axis)  






Classification Model






Class (Flank Tool Wear: `Sharp`, `Used`, `Dulled`)








1 Tool Image






Classification Model






Image Class (Flank Tool Wear: `Sharp`, `Used`, `Dulled`)







 

Future work use cases






Input
Model
Output






[1, ..., 4] Images






Model






Remaining Useful Life (RUL) estimation








[1, ..., 4] Images






Monitoring Model






Fault and Anomaly Detection








[1, ..., 4] Images






Forecasting Model






Multivariate Time Series Prediction








[1, ..., 3] Spectrograms






Model






Signal Drift measurement








[1, ..., 4] Images






Regression Model






Zero-Shot Flank Tool Wear (in µm, 10e-6 meter)








[1, ..., 4] Images






Feature Engineering






Diagnostic Feature Designer







 
description: Sample Count 





 Original Data:  



   Random Split: Train 410 | Validation 54 | Test 48 | Total 512  

   Tool Split:   Train 404 | Validation 52 | Test 56 | Total 512  



 Augmented Labels:  



   Random Split: Train 2460 | Validation 54 | Test 48 | Total 2562  

   Tool Split:   Train 2424 | Validation 52 | Test 56 | Total 2532  





Validation and Test sets are constant for reproducibility and comparison.  

Augmented Dataset (`dataset_aug`): 3072 files  

Augmentation creates 'Aug' prefixed images.


The difference beetween the 2562 and 3072 comes from the fact that the augumented are as well the validaiton and the test images, however thay are not considered in the validation and the test process.


Augumentation:

If you want to test augumentation techniques run `minape/augumentation` scripts. After running the augumentation script the new labels_train_augumented.csv file is created with varying number of the training instances.
description: Data Collection 



We collected the data from the DMG MORI HSC 55 5-axis linear milling machine, using ten shaft milling tools and a C45 tempered steel work-piece, over eight weeks from the Production Centrum.

The signal collecting system comprises a dynamometer (KISTLER 9257B), placed between the machine bench and workpiece holder, an amplifier (KISTLER 5011), a bus-coupler (BECKHOFF EK1100), and the industrial PC (BECKHOFF C6920) for decoding and saving the recorded data.

We have captured three-dimensional time series modalities data using a dynamometer sensor positioned beneath the processed material, which measures forces along three axes (Fx, Fy, Fz) at a high sampling frequency of 10 kHz.The one-dimensional visual modality is the image of the tool taken with a Keyence VHX- S15F profile measurement unit microscope.


The milling phase begins when the tool plunges six millimetres into the workpiece and two millimetres in the radial direction. The milling machine continues constant, slow movement until the tool is outside a workpiece. The spindle head is then lifted and moved to the start position with an offset of two millimetres into the material in the radial direction, and the milling phase is repeated until the tool is destroyed or sparks prevent further machining.


Following each phase of processing, the tool is removed from the milling machine and transported to the measurement stand. Here, utilizing an industrial microscope, a detailed image of the tool is captured. This image facilitates the direct measurement of flank wear, allowing us to categorize each observation into a specific class, as defined by expert-established metrics.

 







Class
Range
Description


Class-1




[0,71)  µm


Sharp


Class-2




[71,110) µm


Used


Class-3




[110,+\infty) µm


Dulled




 





 Naming: T- Tool, R- Ride, B- Blade, e.g. T1R2B3 referes to Tool_1, Ride_2, Blade_3.

10 Tools: T1 - T10  

 Each Tool (T) → 10-15 Rides (R) → 4 Blades (B) per Ride  

Transition: Sharp → Used → Dulled


 
description: Data Structure



The strucuture of the main `Mudestreda` dataset folder:

 

data/

│

├── dataset_aug/

│   ├── spec_x/

│   ├── spec_y/

│   ├── spec_z/

│   └── tool/

│

├── dataset_original/

│   ├── spec_x/

│   ├── spec_y/

│   ├── spec_z/

│   └── tool/

│

├── labels_aug/

│   ├── random_distribution/

│   │   ├── train.csv

│   │   ├── val.csv

│   │   └── test.csv

│   └── tool_distribution/

│       ├── train.csv

│       ├── val.csv

│       └── test.csv

│

├── labels_original/

│   ├── random_distribution/

│   │   ├── train.csv

│   │   ├── val.csv

│   │   └── test.csv

│   └── tool_distribution/

│       ├── train.csv

│       ├── val.csv

│       └── test.csv

│

└── README.md


 
description: Dataset Spliting 



Two types of train/val/test split were used:



Random-based distribution

 Tool-based distributions



 1 Random Distributions

A classic approach with an 80/10/10\% random training/validation/test split.  




 Samples from all 10 tools (T1 - T10) are randomly allocated.  

 Distribution Ratio: Training 80% | Validation 10% | Test 10%








Data Pool Description


T1R1B1, T2R3B2, T3R2B4, T4R1B3, ..., T10R14B4





 




Data Split
Tool #
Sample Size (Original, Augumented)


Training (80%)
Mixed Tools
(410, 2460)


Validation (10%)
Mixed Tools
(54, 54)


Test (10%)
Mixed Tools
(48, 48)





2 Tool-based Distributions

In the Mudestreda Monitoring Dataset, each tool transitions from a sharp to a dull state, represented as a sequence of samples. This approach allows for learning not only the representation of individual samples (tool image and three spectrograms) but also the progression between states. By assigning samples from tools T1-T8 to the training dataset, T9 to validation, and T10 to the test set, without shuffling, the dataset facilitates learning long-term dependencies and state transitions, ideal for models with memory blocks or recurrent neural networks (RNNs).


This tool-based distribution simulates real-life zero-shot scenarios, where the test set, derived from a different tool, challenges the model to generalize its learning to new, unseen conditions. This aspect is crucial for studying zero-shot learning mechanisms and has significant implications for real-world applications, such as predicting tool wear and adapting to new conditions in industrial settings. The dataset offers a unique resource for advancing machine learning, particularly in sequential prediction and understanding zero-shot learning in practical scenarios.




 Each tool represents a sequence: Sharp → Used → Dulled.  

 Samples are divided chronologically without shuffling.  

 Training: Tools T1 - T8 | Validation: Tool T9 | Test: Tool T10  








Samples from Tools T1-T10


T1, T2, ..., T8, T9, T10






 






Data Split
Tool #
Sample Size (Original/Augmented)


Training (80%)
T1 - T8  
404 / 2424      


Validation (10%)
T9  
52 / 52  


Test (10%)
T10    
56 / 56
description: How to use the dataset with Minape repository

Repository link: https://github.com/hubtru/Minape  

As a default notebooks use augmented data with random distribution.

To choose beetween



  original / augumented dataset

  and random / tool distribution follow the instructions.



To choose one of the  option.  Comment (by '#') or uncomment (remove '#') necessary code in the repository `Minape/jupyther notebooks/choose_notebook` in `Dataset acquisition` jupyther notebook cell.

1. Original dataset:



     Preparing the data: Copy the content of the dataset_original folder to './Data' folder with the respect to file stricture described above.      uncomment "Original dataset" part in "Dataset acquisition" cell in jupyter notebooks        comment "Augmented dataset" part in "Dataset acquisition" cell in jupyter notebooks

    Preparing the labels:



         Option 1: Random distrubution



 uncomment "Random distribution" part in "Dataset acquisition" cell

 comment "Tool distribution" part in "Dataset acquisition" cell

 train the uni-modal models from /jupyter notebooks

        combine the models and train the `multi-reccurent_aug_toldist.ipynb`

        evaluate model using `mult-reccurent_aug_tooldist--evaluation.ipynb`



        Option 2: Tool based distribution



        uncomment "Tool distribution" part in "Dataset acquisition" cell

        comment "Random distribution" part in "Dataset acquisition" cell

        train the uni-modal models from /jupyter notebooks

        combine the models and train the `multi-reccurent_aug_toldist.ipynb`

        evaluate model using `mult-reccurent_aug_tooldist--evaluation.ipynb`







2. Augumented dataset



     Preparing the data: Copy the content of the `dataset_aug.7z`` to the './Data/dataset_aug' folder.



        uncomment "Augmented dataset" part in "Dataset acquisition" cell in jupyter notebooks

        comment "Original dataset" part in "Dataset acquisition" cell in jupyter notebooks



     Preparing the labels:



         Option 1: Random distrubution



        uncomment "Random distribution" part in "Dataset acquisition" cell

        comment "Tool distribution" part in "Dataset acquisition" cell

        train the uni-modal models from /jupyter notebooks

        combine the models and train the multi-reccurent_aug_toldist.ipynb

        evaluate model using mult-reccurent_aug_tooldist--evaluation.ipynb



         Option 2: Tool based distribution



        uncomment "Tool distribution" part in "Dataset acquisition" cell

        comment "Random distribution" part in "Dataset acquisition" cell

        train the uni-modal models from /jupyter notebooks

        combine the models and train the multi-reccurent_aug_toldist.ipynb

        evaluate model using mult-reccurent_aug_tooldist--evaluation.ipynb