DOI: 10.5281/zenodo.4421860
title: Wrist-mounted IMU data towards the investigation of in-meal human eating behavior - the Food Intake Cycle (FIC) dataset

description: <strong>Introduction</strong> The Food Intake Cycle (FIC) dataset was created by the Multimedia Understanding Group towards the investigation of <em>in-meal</em> eating behavior. The FIC dataset contains the triaxial acceleration and orientation velocity signals (\(6\) DoF) from \(21\) meal sessions provided by \(12\) unique subjects. All meals were recorded in the restaurant of Aristotle University of Thessaloniki using a commercial smartwatch, the Microsoft Band \(2\)™ for ten out of the twenty-one meals and the Sony Smartwatch \(2\)™ for the remaining meals. In addition, the start and end moments of each food intake cycle as well as of each micromovement are annotated throughout the FIC dataset. <strong>Description</strong> A total of \(12\) subjects were recorded while eating their launch at the university’s cafeteria. The total duration of the \(21\) meals sums up to \(246\) minutes, with a mean duration of \(11.7\) minutes. Each participant was free to select the food of their preference, typically consisting of a starter soup, a salad, a main course and a desert. Prior to the recording, the participant was asked to wear the smartwatch to the hand that he typically uses in his everyday life to manipulate the fork and/or the spoon. A GoPro™ Hero \(5\) camera was already set at the table of the participant using a small, \(23\) cm in height, tripod facing the participant, including both the food tray and upper body part in it’s field of view. The purpose of video recording was to obtain ground truth data by manually annotating the IMU sequences based on the video stream. Participants were also asked to perform a clapping hand movement both at the start and end of the meal, for synchronization purposes (as this movement is distinctive in the accelerometer signal). No other instructions were given to the participants. It should be noted that the FIC dataset does not contain instances related with liquid consumption or eating without the fork, knife and spoon (e.g. eating directly with hands). The accompanying python script <em>viz_dataset.py </em>will visualize the IMU signals and food intake cycle (i.e., bite) ground truth intervals for each of the recordings. Information on how to execute the Python scripts can be found below. <pre><code class="language-python"># The script(s) and the pickle file must be located in the same directory. # Tested with Python 3.6.4 # Requirements: Numpy, Pickle and Matplotlib # Visualize signals and ground truth $ python viz_dataset.py</code></pre> FIC is also tightly related to FreeFIC, a dataset we created in order to investigate the <em>in-the-wild </em>eating behavior. More information on FreeFIC can be found here and here. <strong>Annotation</strong> <em>Micromovements</em> For all recordings, the start and end points of all \(6\) micromovements of interest were manually labeled. The micromovements of interest include: <strong>p</strong>ick food, wrist manipulates a fork to pick food from the plate <strong>u</strong>pwards, wrist moves upwards, towards the mouth area <strong>d</strong>ownwards, wrist moves downwards, away from the mouth area <strong>m</strong>outh, wrist inserts food in mouth <strong>n</strong>o movement, wrist exhibits no movement <strong>o</strong>ther movement, every other wrist movement The annotation process was performed in such a way that the start and end times of each micro-movement span the whole meal session, without overlapping each other. <em>Food intake cycles</em> For all recordings, we annotated the start and end points for each intake cycle (i.e. every bite). Each food intake cycle starts with a <strong>p</strong>, ends with a <strong>d </strong>and contains an <strong>m </strong>micromovement. <strong>Publications</strong> If you plan to use the FIC dataset or any of the resources found in this page, please cite our work: <pre><code>@article{kyritsis2019modeling, title={Modeling Wrist Micromovements to Measure In-Meal Eating Behavior from Inertial Sensor Data}, author={Kyritsis, Konstantinos and Diou, Christos and Delopoulos, Anastasios}, journal={IEEE journal of biomedical and health informatics}, year={2019}, publisher={IEEE}}</code></pre> <pre><code>@inproceedings{kyritsis2017food, title={Food intake detection from inertial sensors using lstm networks}, author={Kyritsis, Konstantinos and Diou, Christos and Delopoulos, Anastasios}, booktitle={International Conference on Image Analysis and Processing}, pages={411--418}, year={2017}, organization={Springer}}</code></pre> <pre><code>@inproceedings{kyritsis2017automated, title={Automated analysis of in meal eating behavior using a commercial wristband IMU sensor}, author={Kyritsis, Konstantinos and Tatli, Christina Lefkothea and Diou, Christos and Delopoulos, Anastasios}, booktitle={2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)}, pages={2843--2846}, year={2017}, organization={IEEE}}</code></pre> <strong>Technical details</strong> We provide the FIC dataset as a pickle. The file can be loaded using Python in the following way: <pre><code class="language-python">import pickle as pkl import numpy as np with open('./FIC.pkl','rb') as fh: dataset = pkl.load(fh)</code></pre> The <em>dataset </em>variable in the snipet above is a dictionary with \(6\) keys. Namely: 'subject_id' 'session_id' 'signals_raw' 'signals_proc' 'meal_gt' 'bite_gt' The contents under a specific key can be obtained by: <pre><code>sub = dataset['subject_id'] # for the subject id ses = dataset['session_id'] # for the session id raw = dataset['signals_raw'] # for the raw IMU signals proc = dataset['signals_proc'] # for the processed IMU signals mm = dataset['mm_gt'] # for the micromovement ground truth bite = dataset['bite_gt'] # for the bite ground truth</code></pre> The <em>sub</em>, <em>ses</em>, <em>raw</em>, <em>proc, mm </em>and<em> gt </em>variables in the snipet above are lists with a length equal to \(21\). Elements across all lists are aligned; e.g., the 3rd element of the list under the 'session_id' key corresponds to the 3rd element of the list under the 'signals_proc' key. <em>sub</em>: list<br> Each element of the sub list is a scalar (integer) that corresponds to the unique identifier of the subject that can take values between \(1\) and \(12\). Moreover, the subject identifier in FIC is in-line with the subject identifier in the FreeFIC dataset (information available here and here); i.e., FIC’s subject with id equal to 2 is the same person as FreeFIC’s subject with id equal to 2. <em>ses: </em>list<br> Each element of this list is a scalar (integer) that corresponds to the unique identifier of the session that can range between 1 and \(3\). It should be noted that not all subjects have the same number of sessions. <em>raw</em>: list<br> Each element of this list is dictionary with the 'acc', 'gyr' and 'offset' keys.<br> The data under the 'acc' key is a \(N_{acc}\times4\) numpy.ndarray that contains the timestamps in seconds (first column) and the <em>\(3D\)</em> raw accelerometer measurements in \(g\) (second, third and forth columns - representing the \(x, y\) and <em>\(z\)</em> axis, respectively). The data under the 'gyr' key is a \(N_{gyr} \times 4\) numpy.ndarray that contains the timestamps in seconds (first column) and the \(3D\) raw gyroscope measurements in \(degrees/second\)(second, third and forth columns - representing the<em> \(x, y\)</em> and \(z\) axis, respectively). All sensor streams are transformed in such a way that reflects all participants wearing the smartwatch at the same hand with the same orientation, thusly achieving data uniformity. This transformation is in par with the signals in the FreeFIC dataset (information available here and here). Finally, the length of the raw accelerometer and gyroscope numpy.ndarrays is different \(N_{acc} \neq N_{gyr}\). This behavior is predictable and is caused by the Android/MS Band platforms. The offset key contains a float that is used to align the IMU sensor streams with the videos that were used for annotation purposes (videos are not provided). <em>proc: </em>list<br> Each element of this list is an \(M \times 7\) numpy.ndarray that contains the timestamps, <em>\(3D\)</em> accelerometer and gyroscope measurements for each meal. Specifically, the first column contains the timestamps in seconds, the second, third and forth columns contain the \(x,y\) and <em>\(z\)</em> accelerometer values in <em>\(g\)</em><strong> </strong>and the fifth, sixth and seventh columns contain the \(x, y\) and <em>\(z\)</em> gyroscope values in <em>\(degrees/second\)</em>. Unlike elements in the <em>raw </em>list, processed measurements (in the <em>proc</em> list) have a constant sampling rate of 100 Hz and the accelerometer/gyroscope measurements are aligned with each other. In addition, all sensor streams are transformed in such a way that reflects all participants wearing the smartwatch at the same hand with the same orientation, thusly achieving data uniformity. This transformation is in par with the signals in the FreeFIC dataset (information available here and here). <em>No other preprocessing is performed on the data</em>; e.g., the acceleration component due to the Earth's gravitational field is present at the processed acceleration measurements. The potential researcher can consult the article "Modeling Wrist Micromovements to Measure In-Meal Eating Behavior from Inertial Sensor Data" by Kyritsis <em>et al. </em>on how to further preprocess the IMU signals (i.e., smooth and remove the gravitational component). <em>mm</em>: list<br> Each element of this list is a \(K \times 3\) numpy.ndarray. Each row represents a single micromovement interval. The first column contains the timestamps of the start moments in seconds, the second column the timestamps of the end moments in seconds and the third column a number representing the type of the micromovement. The identifier to micromovement mapping is provided below:<br> \([1] \rightarrow\) <strong>n</strong>o movement<br> \([2] \rightarrow\) <strong>u</strong>pwards<br> \([3] \rightarrow\) <strong>d</strong>ownwards<br> \([4] \rightarrow\) <strong>p</strong>ick food<br> \([5] \rightarrow\) <strong>m</strong>outh<br> \([6] \rightarrow\) <strong>o</strong>ther movement <em>bite</em>: list<br> Each element of this list is a <strong>\(L\times2\)</strong> numpy.ndarray. Each row represents a single food intake event (i.e., a bite). The first column contains the start moments while the second column contains the end moments of each intake event. Both the start and end moments are provided in seconds. <strong>Ethics and funding</strong> Informed consent, including permission for third-party access to anonymised data, was obtained from all subjects prior to their engagement in the study. The work has received funding from the European Union's Horizon 2020 research and innovation programme under Grant Agreement No 727688 - BigO: Big data against childhood obesity. <strong>Contact</strong> Any inquiries regarding the FIC dataset should be addressed to: Dr. Konstantinos KYRITSIS Multimedia Understanding Group (MUG)<br> Department of Electrical & Computer Engineering<br> Aristotle University of Thessaloniki<br> University Campus, Building C, 3rd floor<br> Thessaloniki, Greece, GR54124 Tel: +30 2310 996359, 996365 <br> Fax: +30 2310 996398<br> E-mail: kokirits [at] mug [dot] ee [dot] auth [dot] gr