Title
Sub Title
AuthorPublisher
Publication year
JtitleJaLC DOI
Abstract
Notes
Genre
URL

Nutrition management based on food recognition and AR measurement
孙, 雅文(Sun, Yawen)
小木, 哲朗(Ogi, Tetsurō)
慶應義塾大学大学院システムデザイン・マネジメント研究科
2019

修士学位論文. 2019年度システムエンジニアリング学 第292号
Thesis or Dissertation
https://koara.lib.keio.ac.jp/xoonips/modules/xoonips/detail.php?koara_id=KO40002001-000020190008

慶應義塾大学学術情報リポジトリ(KOARA)に掲載されているコンテンツの著作権は、それぞれの著作者、学会または出版社/発行者に帰属し、その権利は著作権法によって保護されています。引用にあたっては、著作権法を遵守してご利用ください。
The copyrights of content available on the KeiO Associated Repository of Academic resources (KOARA) belong to the respective authors, academic societies, or
publishers/issuers, and these rights are protected by the Japanese Copyright Act. When quoting the content, please follow the Japanese copyright act.

Powered by TCPDF (www.tcpdf.org)

Master’s Dissertation

2019

Nutrition Management Based on Food
Recognition and AR Measurement
Yawen Sun（Student ID Number：81734570）

Supervisor

Tetsuro Ogi

September 2019

Graduate School of System Design and Management,
Keio University
Major in System Design and Management

SUMMARY OF MASTER’S DISSERTATION
Student
Identification
Number81734570

Name

Yawen Sun

Title
Nutrition Management Based on Food Recognition and AR Measurement

Abstract
With the development of medical, the average life expectancy increases gradually, but
the healthy life expectancy does not increase at the same rate. Plenty of senior citizens
suffered from diseases. In Japan, over 60% people dead from lifestyle diseases every year.
Health management come to be serious nowadays, which is divided into nutrition,
physical activities, and mental health management. It would help people improve healthstates and expand healthy life expectancy. With the widespread use of smartphone, people
are used to using health management applications. But its lack of nutrition management
application, only applications have incomplete functions.
To do nutrition management, a new application was built to help users know thenutrition information of daily foods and drinking. This application could identify foods anddrinking and recall their nutrition information from the system database, the volume offoods and drinking could be calculated, and the calories and other nutrition information canbe converted.
The whole system was divided into three steps, identify, measure and calculate. Basedon machine learning and AR measure, identify and measure would be achieve. And connect
with database, calculation would be completed. Some participates were asked to use thisapplication and answered the questionnaires. After using it, they think it is helpful. But thissystem still has some limits, for no shape foods, it is hard to estimate its nutrition
information. And in future work, new models could be build to identify more food.
In conclusion, this application was developed to help people do nutrition management
and improve health management.
Key Word(5 words)
Nutrition management, Machine learning, AR measure, Core ML and ARKit!2

List of Figure

5

List of Table

7

1. Introduction

8

1.1 Background8

1.1.1 Health Management10

1.1.2 Health Management Applications

11

1.2 Problem

14

1.3 Purpose

16

1.3.1 Improve Health Management16

1.3.2 Nutrition Management Applications

17

2. Related Projects

18

2.1 Related Projects

18

2.1.1 Food Log

18

2.1.2 Food Scanner20

2.1.3 Open Food Facts21

2.2 Difference22

3. Methodology

23

3.1 System

23

3.1.1 Functions

23

3.1.2 Machine Learning25

3.1.2.1 Machine Learning25

3.1.2.2 Advantages of Machine Learning27

3.1.2.3 Core ML

28

3.1.2.4 Core ML Example

30

3.1.2.5 Solution

31

3.1.3 Argument Reality34

3.1.3.1 Argument Reality34

3.1.3.2 ARKit

36

3.1.3.3 AR Measure38

3.1.3.3 ARKit Example

40

3.1.3.4 Solution

41

3.1.4 Segue

43

3.2 System Structure45

4. Verification & Validation

46!3

4.1 Verification & Validation

46

4.1.1 Identity

47

4.1.2 Measure48

4.1.3 Evaluation

49

5. Application

53

5.1 Application

53

6. Observation

55

6.1 Discussion

55

6.1.1. Background Problem

55

6.1.2 Unique Feature55

6.1.3 Application Insufficient56

6.1.4 System Limit

56

6.2 Future work57

7. Conclusion

60

7.1 Conclusion

60

Acknowledgements
Publication List
Reference
Appendix

61
62
63
68

Codes for Application

68

Core ML68

ViewController

73

Line

81

ARSCNView

86

SCNVector3

87

AppDelegate

88

Questionnaire

90!4List of Figure
Figure 1: Average Life and Healthy Life expectancy

8

Figure 2: Japanese Death Cause

9

Figure 3: Apple Application Health

12

Figure 4: Body System

14

Figure 5: Application Calomeal

15

Figure 6: Health Management

16

Figure 7: System Design

17

Figure 8: Application Food Log

18

Figure 9: Food Scanner

20

Figure 10: Open Food Facts

21

Figure 11: Functions Flow

23

Figure 12: Machine Learning

25

Figure 13: Machine Learning

26

Figure 14: Supervised Learning

26

Figure 15: Google Home Mini

27

Figure 16: Core ML Model

28

Figure 17: Core ML

29

Figure 18: Application Digit

30

Figure 19: Import Framework

31

Figure 20: Component outlet

32

Figure 21: Marker-based AR

33

Figure 22: Marker-less AR

35

Figure 23: AR measure principle

38

Figure 24: Anchors

39!5

Figure 25: AR measure principle

39

Figure 26: Application Measure

40

Figure 27: Calculate distance

41

Figure 28: Measure pizza diameter

42

Figure 29: Using Segue to connect to Storyboards

43

Figure 30: System Structure

45

Figure 31: System Design

46

Figure 32: Identity

47

Figure 33: Health applications

49

Figure 34: Calories

50

Figure 35: Machine Learning and AR

51

Figure 36: Application

52

Figure 37: Health Log System

53

Figure 38: Push message

54

Figure 39: Train Model

57

Figure 40: Add database

58!6

List of Table
Table 1: Measure Results

48

Table 2: Pizza nutrition fact

57

!7

1. Introduction
1.1 BackgroundNowadays, with the advances in science and medicine technology, the average life
expectancy increases steadily. However, the healthy life expectancy does not increase at thesame rate. Average life expectancy means the average of how many years people might be
expected to live, and the healthy life expectancy is the average of how many years they might
live in a “healthy” state. Healthy life expectancy is a key summary measure of a population’s
health[1].
World Life Expectancy
Japan Life Expectancy
China Life Expectancy

90

World Healthy Life Expectancy
Japan Healthy Life Expectancy
China Healthy Life Expectancy

81.25

72.5

63.75

55

2000

2005

2010

2015

Figure 1: Average Life and Healthy Life expectancy
As figure 1 states, about 10 years of gap between the average life [2] and healthy life
expectancy[3] around the world. In terms of specific countries, Japan and China, thisdifference still exists. In other words, plenty of people live in unhealthy states, such as seniorcitizens who suffered from the disease in their last few years. One reason for thisphenomenon was that several people are in sub-health status with the fast pace of living!8

rhythm. Usually, people do not mention that they are in sub-health status, but the underlying
disease already exists. Along with the growth of the age, hypo-function on bodies would
induce underlying these diseases.As the population ages and life expectancy increase, healthmanagement should become imperative parts as early as possible in our daily lives. Healthmanagement is a growing industry in the United States. According to the Bureau of Labor
Statistics, from 2010 to 2020 the positions in health management grows to 22 percent[4].

CancerHeart disease

StrokeOthers

30%
42%

16%
12%

Figure 2: Japanese Death Cause
Japan, which’s average life expectancy is over 80 years, is the world's leading longevity
country. However, longevity does not represent that they have a long healthy life. The largest
enemies of a healthy life are lifestyle diseases, which include cancer, heart disease, stroke,liver disease, kidney disease and diabetes[5]. Patients are usually suffering from these diseases
for a long time. As figure 2 has shown [6], in Japan, over 60% of people die because of thesethree diseases, as figure 2 shows.

!91.1.1 Health Management
Health management means health promotion programs, defined by World HealthOrganizations (WHO) which aim to engage and empower individuals and communities tochoose healthy behaviors and reduce the risk of developing chronic diseases and othermorbidities[7]. Health management focuses on addressing and preventing the root causes of ill
health, not only treatment and cure. It would benefit and protect individual people’s healthand quality of life and cover a wide range of society and the environment[8].
Health Japan 21[9], a project of comprehensive implementation of national healthpromotion by the Japanese government. The targets of Health Japan 21 is as following,
achieving extension of healthy life expectancy and reduction of health disparities, theprevention of onset and progression of life-style related diseases, maintenance andimprovement of functions necessary for engaging in social life, putting in place a social
environment to support and protect health and improvement of everyday habit and social
environment relating to nutrition and dietary habits, phasic acuity and exercise[10]. In
consequence, health management is classified into 3 aspects, diet and nutrition, physicalactivity and exercise, and leisure and mental health in Health Japan 21.

!10

1.1.2 Health Management Applications
mHealth (mobile health) is a general term of using the mobile phone and other wirelesstechnology in medical care[11], which is a component of eHealth[12]. The most frequent
mission of mHealth is the use of mobile devices to educate users about preventive healthservices. But it is also applied for disease surveillance, treatment support, epidemic outbreaktracking, and chronic disease management.
Along with the widespread of smartphones and enhancement of health management
consciousness, abundant health management applications appeared in smartphone
applications stores. Health management applications offer health-related services for
smartphones and tablet PCs. Users could use these applications both at home and on-the-go.
Hence health applications become a part of the movement toward mobile health (mHealth)programs. Health management applications are divided into 2 types, one is designed to help
users choose health behavior by offer advice about nutrition and exercise, and the others
could help patients communicate with doctors and doctors to keep accurate records ofpatients' health status[13].
For example, the iOS system, one of the predominant smartphone systems, has its
application called Health[14]. This application could help users learn about their health and. It
centralizes health data from iPhone, Apple Watch and third-party applications, and it has fourkeys, which are activity,

sleep, mindfulness, and nutrition, as figure 3 shows. Based on

ResearchKit and CareKit, the user could communicate with doctors. ResearchKit is a tool tohelp medical research obtain health data from iPhone, users could take part and allow theirdata to be used in meaningful studies. And CareKit could help users track users’ symptoms
and medications and share this information with the care team.

!11

Figure 3: Apple Application Health!12

Health could record several physical activity data to calculate how much users exercise
and users’ all-day calorie burn, such as the total walking steps and distances every day. With
Apple Watch, the functions were expanded, the heart rate could be measured automatically.
In terms of leisure and mental health aspects, good sleeping could help people restore andrepair the body. Compare with third-party applications, which could monitor user's sleep
status, these applications not only record sleeping times but also could distinguish users
whether they enter deep sleeping to determine sleeping quality.
Health also could count users' carbs, calories or a host of other important nutritional
metrics with third-party applications. These third-party applications could help users keep acloser eye on every meal by displaying all the nutritional data. Nutritional data covers various
aspects, dietary cholesterol, dietary energy, dietary sugar, caffeine, calcium, iron, water,
vitamin, etc.. So it could provide all-around nutrition management to users.

!13

1.2 Problem
As mentioned above, people already have a realize of health management and do it withsmartphone applications gradually. Considering a human being's body as a system, theapplications which examine physical activities data and sleeping status record the output datafrom human bodies. On the contrary, the nutrition data in input data into human bodies, as
figure 4 shows. This is not easy to obtain since current technology cannot monitor thedigestive system at any time. Instead of monitoring the digestive system, recording daily foodand drinks nutrition data is an acceptable method to do nutrition management.

Figure 4: Body System
Though serval third-party nutrition applications are in application stores, the functions ofthem are single and incomplete. For example, an application called Calomeal [16]. Thisapplication, developed by Life Log Technology, Inc, aims to extend human beings' healthy
lives from diet aspect. In Japan, with low birthrate and aging, labor people decline and social
insurance becomes the main issue. In terms of individuals, the health problem is a serious
mission. The main cause of Japanese death, which is related to three major diseases, is an!14unhealthy lifestyle. And the extensive factors of lifestyle-related diseases are diet, sleeping,smoking, movement, and exercise. Calomeal supports users' record life log to have a healthy
life with machine learning, as figure 5 shows.
With Calomeal, users take photos for daily foods drinks firstly. The application could
identify food and drinks by AI. Based on the result of AI analysis, the system would indicate
nutrition information about food and drinks. As a result, the calories and nutrition information
about daily meals are recorded. Besides, this application assists users to manage weight andactivities. The nutrition information from this application is a reference value since thevolume of foods is unknown.

Figure 5: Application Calomeal
Similar problems also happened in other related applications. Nutrition information about
daily food and drink cannot be recorded accurately, and nutrition management has many
imperfections.
!15

1.3 Purpose
1.3.1 Improve Health Management
Health not only means the absence of disease, but also means satisfactory conditions ofquality, physical, social, intellectual, emotional, and spiritual. Health management is asignificant project for our daily life, this study intends to provide human beings with morecomprehensive health management, to improve the health states and expand healthy life
expectancy. As above mentioned, health management could be enhanced by several aspects,
this study would focus on nutrition management, as figure 6 shows.

Figure 6: Health Management
Nutrition is the cornerstone of good health and the cutting edge of prevention, which
provides all nutrients in both kind and amount[16]. The nutrients from the food we eat, are themost essential continuing environmental factors affecting human beings' growth,development, functional abilities, and health.
Nutrition management, which defined as assisting with or providing a balanced dietaryintake of foods and fluids, is one of the best approaches to prevent and control healthproblems such as heart disease, high blood pressure and some types of cancers.

!16

1.3.2 Nutrition Management Applications
As we know, to do nutrition management, recording the nutrition information of daily
foods and drinking is a convenient method. To know the nutrient content of a food, you needto measure the nutrient content of each food and the number of servings of food. In the caseof pizza, the same size of Hawaiian pizza and Margaret pizza have different calories. At thesame time, the amount of pizza intake will also affect the amount of calories people ingest. To
record this information, starting with measuring the volume of foods and drinking and thenconvert into weight and nutrition information would be a convenient approach.
Figure 7 shows the concept of the proposed system, to get the calories and other nutrition
information, the whole system is divided into three steps. In the first step, this application
could identify foods or drinks. And in the second step, the system would measure the volumeof them. The method of measurement is also related to the type of food and drink. Lastly, thesystem calculate foods and drinking weight and nutrition data based on volume. If thenutrition data user intake every day is known, nutrition management could be done simply.

Figure 7: System Design!17

2. Related Projects
2.1 Related Projects
2.1.1 Food Log
Food Log[37] is a project managed by Aizawa at the University of Tokyo, which is amultimedia food-recording tool that provides a newfangled approach for recording daily foodintake primarily for healthcare purposes, as figure 8 displays. It's a novel use of imageprocessing techniques presents significant potential for the development of new healthcare
monitoring applications[17].

Figure 8: Application Food Log!18

Food is an attractive topic in the multimedia community, it is indispensable for daily lives
and a significant part of fields such as healthcare, agriculture, and food marketing andproduction. This project focus on investigating the capture, processing, and use of multimedia
data related to people's daily food intake, to the health and quality of human being lives.
FoodLog, a multimedia food-recording application, provides a fresh method for recording
users' daily food and drinks intake primarily for health management purposes.

!192.1.2 Food Scanner
Food scanner[19], a project from Fraunhofer, is a food lab insider mobile phone. Thissystem could analyze food and also access to quantity and nutrition facts.
The system in the smartphone will access users to choose quality food and also provide
individual nutritional and fitness advice to users, as figure 9 shows. According to the volume
from 3D triangulation, the system would provide real nutrition information about food. It also
could connect with other applications to get the user’s specific data, such as activity data.
And then nutrition and consumption can be evaluated and a recommendation for the user canbe provided.

Figure 9: Food Scanner
According to the environmental organization WWF Germany[20], ten million metric tons offood, which are edible, are thrown in the garbage every year in Germany. Food scanner will
support consumers and supermarket operators to determine the quality of foods. The ripenessand shelf life of foods is determined with infrared measurements and the results are passed toapplications in the smartphone.

!202.1.3 Open Food Facts
Open Food Facts [21] is a database of food products with ingredients, allergens, nutrition
facts and all the tidbits of information users could find on product labels, as figure 10 shows.
In this project, everyone could be contributors, who use the applications [22] in smartphones toscan barcodes and upload pictures of products and their labels. Meanwhile, this database isfor everyone, a database is published as open data and can be reused by anyone and for any
use.

Figure 10: Open Food Facts
This application, which aims to make better food choices, could help the user understandthe fine print on product labels, such as food additives E-numbers, allergens, packaging
codes, and the nutritional quality of each product. It could help users find the product thatmatches their criteria. Users also could make comparisons from different brands' products tochange for healthier cereals. This database also could be used for thought, innovation, andscience.

!21

2.2 Difference
The application in this project focus on measuring the size of food and drinking to get thevolume of food and drinking. According to volume, the weight and nutrition information
would be estimated. So the key to the application is using AR to measure foods and drinking.
Food Log is an application that could record photos of users' daily foods and drinks, but it
does not record the nutrition data of them. But only pictures are not enough for people to do
nutrition management, people need to know how much calories or other nutrition data they
intake every day. From the application in this research, people could know this information to
do nutrition management better.
Food scan could provide food nutrition information with 3D triangulation, but it needs
additional devices. The application in this study could provide food nutrition information
without any additional device. Besides, food scan is a project, which focuses on determining
the quality of foods in the market, to help the consumer or the market whether the need tothrow food to reduce unnecessary waste. The core purpose of a food scan is not helping
people do nutrition management, providing nutrition information is just associated functionsin the application.
Open Food Facts is a database of foods and drinking, which could help users know thenutrition facts of the food. The food with wrapping with labels could be provided withnutrition information easily. But for the food without wrapping and labels, such as foods inthe restaurant, Open Food Facts cannot provide the nutrition information. The application inthis study breaks in this limit, it could identify foods and drinking with machine learning andmeasure the volume of them.

!22

3. Methodology
3.1 System
3.1.1 Functions
Nutrition management is one parts of health management, and this system has 3 mainfunctions. The main functions of the application in this study are identifying foods or
drinking and measuring their size and calculating weight and nutritions. The functions flow
chart is as figure 11 shows. With Core ML, the framework of Apple for machine learning, theapplication could identify foods and drinking. And then the system would recall theinformation from the database of these foods or drinking, and based on the measured resultwith the framework ARKit, the system would calculate the volume of foods and drinking.
Based on the volume, the weight could be estimated. And if the weigh is known, the calories
and other nutrition information also could be converted with nutrition database. The
applications in this study could concentrate on these three functions.

Figure 11: Functions Flow!23

Afterward, the application would save everyday data and cooperate with otherapplications, combining with information of physical activities and mental health, to get more
health information from users. According to comprehensive health information from several
health applications, the system could do health management more preferably and provide
users more suitable healthy advice.

!24

3.1.2 Machine Learning
3.1.2.1 Machine Learning
Machine Learning is an application of artificial intelligence (AI), lying at the intersectionof computer science and statistics[42], as figure12 shows, it could improve system
automatically from experience without any programming and focus on the development of acomputer that can access data and use it learn for themselves[23].

Figure 12: Machine Learning
The process of learning starts with observations or data, which is aim to look for patterns
in data and make better decisions in the future based on the examples, it allows the computers
to learn automatically without human intervention or assistance and adjust actions
accordingly.
Machine learning can analysis of massive quantities of data. It could deliver faster, moreaccurate decisions to identify profitable opportunities or avoid dangerous risks, but it also
needs additional time and resources to train.
Machine learning could be divided into classified as supervised, unsupervised andreinforcement[24], as figure 13 shows. For supervised learning, the data set is given and theoutput is also known, it was classified into regression and classification, as figure14 states. It
could be applied to cancer prediction[43]. Unsupervised learning, as the name suggests, the!25

Figure 13: Machine Learning

Figure 14: Supervised Learning
data set isalso given, but the label of data set is unknown. Developers should find thestructures from the data set. It is divided into clustering and dimensionality reduction.
Reinforcement learning means taking applicable actions to obtain maximum rewards in aparticular situation. So the software and machines optimize operation in the specific situation.
The difference of reinforcement learning is that it decides what to do to perform in thespecific task and does not has the correct answer. But supervised learning focus on finding
the correct answer.
!26

3.1.2.2 Advantages of Machine Learning
Machine learning could identify trends and patterns easily and do not need humanintervention. Since it has handling multi-dimensional and multi-vanity data and this data
could be added continuously, the system also could be improved continuously. Meanwhile, it
is also able to apply in applications widely[25].

Figure 15: Google Home Mini
One of the machine learning applications is virtual personal assistants, which assist in
finding information, when asked over voice, such as Amazon Echo and Google Home[44], asfigure 15 shows. Machine learning is a signification part of these virtual personal assistants,
thee could accumulate and improve the information based on previous effects with them.
According to this data, the system would provide better information which is tailored tousers’ preferences.
In terms of the application in this study, based on the original database, the system could
identify several foods and drink. And with the mobilization of the applications, the system
would collect more images of foods and drinking and extend the database at the same time.
That is the system upgrade itself without manual work.

!27

3.1.2.3 Core ML
Core ML[26] is a foundation for the domain-specific framework for machine learning fromApple, which is based on Swift. Developers could use Core Ml to integrate machine learningmodels into applications, and it provides a consolidated representation for all models. The
application could use Core ML APIS and user data to make predictions and to train or finetune models on the device of users.

Figure 16: Core ML Model
As the figure 16 shows, the flowing is from left to right. Core ML model is a result, which
was applied to a machine learning algorithm to a set of training data. Based on new input
data, the model could make predictions. The model also could complete various tasks, which
are difficult or impossible to write in code. For example, the model could detect specificobjects from photos with its pixels.
The key to Core ML is Core ML model. Developers could download several Core ML
models from Apple's official website. The models used in this research are called
SqueezeNet, which is a small deep neural network architecture, it could classify the dominant
object in a camera frame or image.
Furthermore, developers could build their models to match their needs with Create ML[27].
With Create ML, developers could their database for models[45]. In this process, plenty of
pictures are added into the database, and the system would “remember” these objects withpictures. This process likes humans to learn new knowledge or skills, “adding pictures” is!28

familiar with the practice process. So the database could be updated easily and quickly. With
these methods, the application in this study could identify more kinds of foods, evendifferentiate similar foods.
Leveraging the CPU, GPU, and Neural Engine, Core ML could optimize on-device
performance, and minimize the memory footprint and power consumption. A network
connection is not necessary when running the model, this point would protect the privacy of
the user’s data and application responsive.

Figure 17: Core ML
Core ML builds on top of low-level primitives, such as Accelerate, BNNS, and Metal
Performance Shaders, it could support Vision for analyzing images, Natural Language forprocessing text, Speech for converting audio to text and SoundAnalysis for identifying
sounds in audio, as figure 17 shows. The application in this study uses the function ofanalyzing images to identify foods.

!293.1.2.4 Core ML Example
Core ML could be applied into various applications, since it has well-made functions. One
example of Core ML applications is using it to build an application which could recognizes
hand-drawn digits[39]. The basic steps for integrating a Core ML model into iOS application
with Core Ml is as figure 18.

Figure 18: Application Digit

It the user draw the number in the screen by hand. As we know, the handwritten number isquite different every time, which is not that standard. But the system could recognize it anddetermine which is match the question. It is a basic functions of Core ML, recognizing theobjects. In this process, with the handwritten number in the screen increasing,!30

3.1.2.5 Solution
Core ML is framework, which cannot be used directly by users, to apply Core ML and
Vision in iOS device, some program process is necessary. And programming would be donein Xcode, an integrated development environment for macOS, it also includes a set ofsoftware development tools developed by Apple for developing software for Mac OS, iOS,
iPadOS, watchOS, and tvOS. So the pre requisites for this process are MacOS, Xcode 9 orabove and devices with iOS 11 or above.
Firstly, a Xcode project was created, Xcode has many preset. To create machine learning
application, single application was selected. Meanwhile , the programming language should
be swift.
To create the user interface, Xcode has storyboard to do is easily, which is a visual
representation of the user interface of an iOS application, it shows screens of content and theconnection between screens. Each storyboard represents a set of scenes, such as viewcontroller and its views. And these view controllers could be connected by segue objects. So
in this application storyboard including the following components, UIButtons, UIImage andUILabel. UIButtons were used to input photos, UIImages was used to display input photos
and UILabel was applied to display the name of foods or drinks.

Figure 19: Import framework
Meanwhile, in swift file, the project needs import UIKit. AVFoundation, Core ML and
Vision framework would be imported in later steps, as figure 19 shows.

!31

Figure 20: Component outlet
After adding components in storyboard, some components were outlet to programming
editor, as figure 20 shows. IBAction in programming editor outlet for each UIButton,
UILabel outlet for the label to show identify results. And UIImage outlet for the image view.
Next, to use the UIImagePickerController(), class should be extended with
UINavigationControllerDelegate and UIImagePickerControllerDelegate.
This application needs to access to iPhone camera and photo for predictions, the key inprivacy should be added, Camera Usage Description and Value. The initial setup wascompleted by this step.
And then adding Core ML model to the project, Apple has some open source model could
be download in office website. After downloading, the model was dragged and dropped into
the Xcode project structure. The input and output parameters and other information of model
should be checked. The information of model showed the input the model takes in and the!32

output the model returns. In this project, it takes in a 299x299 images, and would return withthe most like category, and the probability of each category.
Clicking the arrow in front of the model label in the Model Class section to open themodel class, Xcode would generate for Core ML model automagically. To get the expected
prediction as output from the model, the expected input should be given. Importing Core ML
and the Vision framework into ViewController.Swift file to convert the image.
To using Core ML, some code are necessary. A generate method to use Core ML model isprediction(image:), it is used to predict the object in the given image. With pixelBuffer
variable, the size of image could be resized. Once the prediction is return, which is String
type. The classifier label would be updated to set its text to what it has been recognized.
The machine learning parts process was as above. The project could identify the foods anddrinks, and pass these information to the next view controller, ARKit, with segue.

!33

3.1.3 Argument Reality
3.1.3.1 Argument Reality
Argument reality (AR) means the integration of digital information with the user's
environment in real-time[46]. Different from virtual reality (VR), which creates an artificial
environment, AR adds new information into the existing environment.
The main approaches to enable AR works are simultaneous localization and mapping(SLAM), recognition based and location-based[28]. SLAM is the most effective approach topresent virtual objects in real-world objects, it localizes sensors concerning theirsurroundings, and at the same time map the structure of the environment. SLAM is aim tosolve simultaneous localization and mapping problem and now every augmented reality
development kit could provide SLAM functionality. Recognition based means using a camera
to recognize visual markers or objects to present a cover when the marker is observed by the

Figure 21: Marker-based AR!34

device, as figure 21 shows. Location-based, different from recognition base, is based on GPS
digital compass, velocity meter, or accelerometer to provide location information and input
augmented reality visualizations. It is also called marker-less augmented reality, as figure 22
shows.

Figure 22: Marker-less AR
As above figure 19[41] and 20 show, Marker-based AR recognizes a simple marker anddisplays the real things in the screens. But for marker-less AR, it could recognizes object
without any markers and meanwhile add some virtual information into real environment.
In AR applications, a live, onscreen view of the physical world is presented by the camera.
And on this view, three-dimensional virtual objects are superimposed to create the illusionthat they exist veritably.
!35

3.1.3.2 ARKit
ARKit[29] is another framework of Apple to produce augmented reality experiences inapplications by integrating iOS device camera and motion tracking functions. ARKit could
blend virtual objects with the real world seamlessly to deliver impressively and engage
experience.
Creating and tracking a correspondence between the real-world space the users inhabits
and a virtual space where users can model visual content is the basic requirement for AR andthe defining feature of ARKit[38]. To create this relationship, ARKit uses a technology called
Visual-Inertial Odometry.
ARKit combines the benefit of device motion tracking, camera scene capture, advanced
scene processing, and display conveniences, which could simplify the processor building an
AR experience. In WWDC 2019, ARKit 3 was released[40], now it has the following
functions, people occlusion, motion capture, simultaneous front and back camera, multiple
face tracking, collaborative sessions. Among these functions, People occlusion, a new
function of AR, means AR content could pass behind and in front of people in the real world,
it would make AR experiences more immersive and enable green screen-style effects inalmost any environment at the same time. Motion capture means the system could understandhuman body position and movement, it could track joints and bones. So it would provide a
new AR experiences involved with people and objects at the same time. ARKit 3 was also
reformed in other aspects, it could detect up to 100 images at a time, in complex environment
it could detect 3D-object more robust. For facial recognition, it would use front and backcamera simultaneously, which allow users interact with AR content in new ways, with new
iPhone device it can track up to three faces at once.

!36

ARKit could track motion fast and stable since VIO blends CoreMotion data and camera
sensor data. These two inputs allows the system know how device move more accurate
without any need for additional calibration. It also makes object look like being kept in actual
environment, rather than hovering over it. ARKit also makes full use of the camera sensor todecide the amount of light present in a scene and apply the idealize amount of lighting tovirtual objects. ARKit has high-performance hardware and allows impressive levels of detailand visual fidelity since it was supported by Scenekit, Metal and third-party tools[30].

!373.1.3.3 AR Measure
AR measurement is one function of AR, which could gauge the size of real-world objects
in real-time. The principle of AR measure is that the captured frames of the camera areprocessed by the program, and computer vision tries to identify the track some relevant points
in the images, figure 23 shows the flow charts. The tracked points are called anchors[31], asfigure 24 shows. These anchors were identified, and as device moves, the perspective of thecamera is also moved. The position of anchors is updated, as figure 25 shows. With anamount of processing, the distance could be estimated.

Figure 23: AR measure principle!38

Figure 24: Anchors

Figure 25: AR measure Principle!39

3.1.3.3 ARKit Example
Apple released an application called Measure[32] in September 2018, which could gauge
the size of real-world objects quickly, detect the dimensions of rectangular objects
automatically with the iOS device camera, and also save measure result to a photo, as figure
26 shows.
On the subject of the application in this research, AR measure could help users get thevolume of foods and drinks in real-time.

Figure 26: Application Measure!40

3.1.3.4 Solution
An ARKit project was created to measure the size of foods and drinking. When creating a
new project, this time using the “Augmented Reality App” template. Except ARKit, thisproject would also use SceneKit for its 3D and 2D feature.
Camera Usage Description in privacy still need to be added to this project, sinceaugmented reality requires the use of a camera.
A new swift file was created, in the measure view controller file, UIKit, SceneKit and
ARKit were imported with the ARSCNViewDelegate, which provide SceneKit content
corresponding to ARAnchor objects tracked by the view’s AR session and manage the view’s
automatic updating of such content.
In storyboard, add a new view controller, and ARKit SceneView object. To set up the
SceneView, add the IBOutlet to the SceneView object in the storyboard file, create a set scenefunction and set up the ARSCNViewDelegate. And then set up the ARSession, create aworld-tracking

sessionwithhorizontal

plane

detectionandimplement

the

ARSCNViewDelegate function. Then create a sphere.

Figure 27: Calculate distance
To measure the distance between two points, an extension to SCNVector3 was added for
calculating the distance between 2 vectors, as figure 27 shows. Calculate the distance
between two points and draw a line between them, add the extension to eh SCNNode withfunctions to achieve it. And the UI to display the distance and a point to recognize the centerof the screen.
!41

A new SCNNode class was created, the distance would be presented as a node in the
SceneView. And adding the TextNode to the SceneView. And the measure result would beget. For example, figure 28 shows the diameter of the pizza.

Figure 28: Measure pizza diameter!42

3.1.4 Segue
Segue[33], which is a UIKit prepares for and performs the visual transition between twoview controllers[34], defines the flow of the application’s interface. Segue defines a transition
between two view controllers in the application storyboard file. The starting point of Segue
could be a button, table row, or gesture recognizer that initiates the segue, and the endpoint
would be the view controller should be displayed, as figure 29 shows.

Figure 29: Using Segue to connect to Storyboards
Segue has four different types, show, show detail, present modally and presents as
Popover. In this application, the type is shown, which displays the new content using theshowViewController:sender: method of the target view controller. For most view controllers,
this segue presents the new content modally over the source view controller.

!43

In terms of this application, with segue, Core ML and ARKit, which are in two different
view controller, could be combined in one application. These two view controllers werecreated independently before. And then open ARKit view controller as initial, and add Core
ML view controller into storyboard and two new buttons into each view controllers as
connect buttons. When the object was identified by Core ML, segue passes this information
to ARKit to her system to calculate the calorie and nutrition data. So after identifying, users
need to click up the connect button, the screen would jump to measure screen, then users
could measure the size of food.

!44

3.2 System Structure
Considering the application as a whole system, and its elements could be divided into two
types, software and hardwares. For hardwares, it has iPhone and camera on it. For software, it
has two frameworks, Core ML and ARKit. These elements are independent objects, but they
have some connections for each other to contribute the application.
The system structure of this application is simple, base on hardware iPhone and its
camera, camera is one parts of iPhone. Swift is the programming language for thisapplication, which is the basic of software part. Core ML and ARKit are two frameworks of
Swift, one is for identify and the other is for measurement. To achieve the functions of these
two frameworks, the camera would be used. Core ML and ARKit are two independent
frameworks, this application uses Segue to combine these two functions, which is a UIKit ofApple. With these connection, the system could implement integral functions, as figure 30
shows.

Figure 30: System Structure!45

4. Verification & Validation
4.1 Verification & Validation
Verification, a process to proof of compliance with specifications, determines whether or
not a product meets the requirements or specifications. In this system, identity and measureboth should be verified.
Validation, a process to proof user satisfaction, means the assessment of whether theplanned or delivered system fulfills the sponsor's operational need in the realistic
environment.

Figure 31: System Design
Verification and validation support the development process, ensure the developer provers
credible results and satires the customers. The main techniques of verification and validation
are inspection, analysis, demonstration and test, and to collect data for verification andvalidation, questionnaire, interview and observation are common methods.
!46

In this system, verification means verifying the functions of identifying and measure, andvalidation means this application whether meets users’ needs, which could help users get thenutrition information quickly and convenient, as figure 31 shows.

4.1.1 Identity
Using this application to identify several kinds of objects, such as bottled water and pizza.
The system could identify them correctly, and it also indicates how much percent it matches
to result. Figure 32 shows the results, the result of pizza is 91.49%, the accuracy of which isquite high. The result of measurement of a bottle of water is 77.22%, which is also very
accurate. Most food objects could be identified and the probability could be calculatedscientifically. As the number and types of objects increase, the ability of recognizing theobjects will be stronger due to the machine learning function.

Figure 32: Identity!47

4.1.2 Measure
Verifying the measure function means verifying the results from the AR measurement partwhether close to real value. So comparing the measured result from the real rule and AR
measure system is a convenient approach. Table 1 shows three objects are used to evaluate
the system: bottle water, pizza and doughnut. The size and shape of them are quite different.

Bottle Water

Pizza

Doughnut

20.14 cm

23.00 cm

7.9 cmFirst time

19.13 cm

22.25 cm

7.4 cmSecond time

21.46 cm

21.95 cm

8.2 cmThird time

19.27 cm

21.75 cm

7.2 cmAverage19.95 cm

21.98 cm

7.6 cmStandard deviation

1.31

0.25

0.53

Error rate

1%

4%

4%

Real ruler

AR measure

Table 1: Measure Results
From the value of average and standard deviation in the table, we could find that despite
the results from the AR measure system are different every time, but they are stable aroundthe real size. Compared with the results from the real rule, the error rate is under 5%, whichmeans the result from the AR measure system is reliable. Just providing the weight of thefood is not enough for users to know the actual calories of the food. The calories of the foodis provided by the time the weight or size is measured. In the experiment, the size, weight andcalories of a pizza are measured and showed to the user. By comparing with the reference
calories, we found that the calorie calculated by the system is very close to the true value.
Calculating food calories through the system is reliable.

!484.1.3 Evaluation
In this research evaluation, 20 participants took part in the survey, the gender ratio of theexperimental participants is reasonable. Before the questionnaires, the tutorial of how to use
this application was shown to participants, and then the participants were asked to do it bythemselves. All the answers collected by the participants was based on the experience ofusing the application.
Here are some analysis results from the evaluation.
With the development of the smartphone, participants started to use smartphone
applications to manage their health. Among these applications, they are willing to use
applications to record their activities, as figure 48 shows, such as everyday steps. They often
use motion detection software to record their movements, such as exercise time and heartrate. Besides, some participants have used software to detect sleep. In conversations withfemale participants, we also learned that some women use software to record their
physiological cycles, such as menstrual time, ovulation, etc. This type of health software isalso very popular.

5

Do you use health apps
Do you use apps to record activities
Do you use apps to monitor sleep

3.75

2.5

1.25

0

Average

Figure 33: Health applications!49

From the analysis results, figure 34, it is obvious that participants care about foods or drink
calories but they seldom do the calculation for calories. There may be many reasons for thisphenomenon, and we learned a few main reasons for the participants. Some participants
reported that it is very difficult to calculate calories for food because there are different kinds
of foods. Some participants mentioned that since no convenience tools could be used to getcalories, it is not easy to get nutrition information for food and drinking. From the gap
between the two fold lines, we can also think that for people who pay attention to calories, itis necessary for them to know the clear and accurate calorie information.

Do you care about calories

Do you calculate foods calories

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Average0

2.5

5

Figure 34: Calories

!50

7.5

10

Besides, most of the participants know what is machine learning and AR. However, they
seldom use machine learning and AR-related applications. Also, there is a possibility thatthey do not know some of the applications they are using are based on machine learning andAR technology, as figure 35 shows.
Do you know Machine Learning or AR
Did you use ML or AR apps

5

3.75

2.5

1.25

0

Average

Figure 35: Machine Learning and AR
After trying this application, participants think this application is convenience and theinformation in this application is helpful for them to do nutrition and health management.
Many participants said that if the software is developed successfully, they are willing to
download this software.

!51

Figure 36 shows that the conclusion could be drawn from the survey that the participants
concern about the calories of food in daily life and want to know the number of calories they
consume every day. Since calorie calculations are not easy, they are usually calculated by thecalorie content and the amount of food consumed per meal. The measurement of food weightand size is not very easy in daily life, so many people are not willing to calculate bythemselves. A software that can simply show calories is very useful.

Do you think this app is convenience
Do you think the information in app is helpful
Do you think it helps nutrition management
Do you think it helps health management
5

3.75

2.5

1.25

0

Average
Figure 36: Application!52

5. Application
5.1 Application
Except for its functions, the system also could be applied to other health management
projects. For instance, there is one project called Health Log, which is focused on pushingmessages to users to keep health. The application is related to a database, which includes user
health condition, time, location and steps count, etc.. And according to this information, it
could push personalized messages to users, as figure 37 and 38 shows.
Connecting with these applications, the nutrition information could be added into the
database. If users do not intake apposite nutrition, it could remind users to eat more or less.
For example, for people with high blood pressure, lack of exercise, and low-temperaturesensitivity, calorie intake standards are different. For people who are under-exercised, asmaller amount of calories will not cause fat accumulation. In addition, the software can push
different messages to the user by reading the calorie value of the user's daily intake of food. If
the user eats a lot of high-calorie food today but has fewer steps, the software can pushmessages to the user to take more steps or exercise.

Figure 37: Health Log System!53

At the same time, because the Health Log application divides users into groups withdifferent characteristics, the software in this article can recommend foods suitable for their
physical condition to different users. For example, for people with high blood pressure,excessive exposure to high-calorie foods may be detrimental to good health, and the software
will recommend some healthy foods that are good for fruits and vegetables andcardiovascular and cardiovascular health. Or if the user wants to take some high-calorie
foods, the software can strictly control the amount of food.
Combining with other health data, the system could judge users' health status and push
better messages to users.

Figure 38: Push message!54

6. Observation
6.1 Discussion
6.1.1. Background Problem
As background part mentioned, healthy life expectancy problem became serious inmodern society, plenty of people suffer from disease when they are old. Meanwhile lifestylediseases, cancer, heart disease and stroke, become the main reason for Japanese death causes.
So a healthy good lifestyle become significant for people’s life.
This application would help users know the calories and other nutrition information offood to do nutrition management. Nutrition management is one of imperative parts of healthmanagement. If health management is proved, in the other words a health lifestyle is adopted,
the health state would also improved. So healthy life expectancy would increase and lifestyledisease would reduce as also, the problems mentioned in the background part would be
solved.

6.1.2 Unique Feature
The application could identify food and drinking and calculate the calories and nutrition
information at real time, which is convenience for user to do nutrition management. Comparewith other nutrition management application, it provide a reference value to users, which ismore closed to the real value, since the calories and nutrition value is based on the measure
results from AR measure.
Many measure applications are in application market, but most of them provide thedirectly measure results. But this application do some convert for measure results connectedwith nutrition database. It presents users a comprehensive and favorable information to users.
It is also the innovation of this application.

!55

Besides, this application combines two frontier technology, machine learning and AR, andmake full use of their function and advantages. Usually they are used in dependent
application for single function. Actually these two technology have some common points, forexample, in this application they both use some image-process functions. So once theapplication obtain the authority of camera and photos, the application could collect the
information of foods and drinking quickly.

6.1.3 Application Insufficient
However, by now this application still has many defects. For example, in this application,
machine learning and AR measure are in two screens. So after identify, the interfaces needjump to the next one, which is not succinct. Ideally, machine learning and AR measure are in
one screen, after identify, the system could get into measure process automatically.
Additionally, the project does not have its own Core ML models and comprehensive
database. It means that the application only can identify and measure several kinds of foods
and drinks. Building a specific machine learning model and database would enhancement
utility.

6.1.4 System Limit
Sine in this system the nutrition information are from volume, and the volume are from
AR measure. Once the food or drinks is irregular, such as Mapo Tofu, which means the shapeis not easy to measure and calculate, the system could not provide nutrition information. To
solve this problem, 3D scanner would be applied.

!56

6.2 Future work
Since the measured result from AR is a rough value, the calories and nutrition value fromthis application is just reference value. The reference value should close to real value as close
as possible. One of the methods, which improves accuracy, is that making the system identify
more kinds of foods.
Pizza

ServingCalories

Mushroom Pizza

1 slice (288g)

611 calBeef Pizza

1 slice (235g)

714 calVegetable Pizza

1 slice (156g)

399 calHawaiian Pizza

1 slice (134g)

154 calTable 2: Pizza nutrition fact
Identifying more kinds of foods not only means identify more foods, but also could
classify similar foods. For example, pizza has amounts of kinds, different kinds have different
calories and nutrition value. Table 2 shows the nutrition fact of different pizza[35], it isobvious that vegetable pizza has fewer calories than meat series pizza.

Figure 39: Train Model!57

To classify these pizzas and get more accurate results, new Core ML models could betrained, which has a larger database of food and drinks. The process of training Core ML
models is as following flow chart, as figure 39[36] shows.
Preparing data is the first step of training, which means to prepare plenty of images offoods and drinks, as figure 40 shows. And then creating a training data set from 80% of foodimages for each label, and reaming 20% of images for testing. During this process, any given
image should appear in only one data set or the other. Then adding images into Training Data
and Testing Data folders, and creating labels. In each labels, at least 10 images for the
training set, the more the better. But the number of images for each label should be balanced.
The format and size of images are no limit, but angles and lighting conditions of images
should be different. Preparing data is the most significant step for training models.

Figure 40: Add database!58

Creating an image classifier project means using Create ML to create an image
classification project in Xcode and then drag the Training Data folder and Testing Data folder
onto the project, and then staring the training process. During the training process, the model
would learn from the training set iteratively and checks accuracy with the validations set, thedata of these two settings is random. So the results of each time training are different. After
training, the system would report training and validation accuracy scores.
In the testing process, the system would use the image from the Testing Data folder, which
has never seen before. The system would test all of the images and present the overall tasing
accuracy. If the evaluation performance is not good enough, training data should be changed.
If the model performs well enough, the model could be saved and used into applications.

!59

7. Conclusion
7.1 Conclusion
A new application was builded to support users do nutrition management and improvehealth management. This application could identify foods and drinking, and measure thevolume of foods and drinking to obtain the calories and other nutrition information of them.
In order to accomplish the functions of identify and measure, Apple’s frameworks Core
ML and ARKit were applied into the system. Core ML is a framework for machine learningin the iOS devices. Based on Core ML models, the system could classify various objects withcamera in iPhone or photos. The models of Core ML are able to train to classify more objects.
ARKit is a framework for AR, measure is one of significant AR functions, which could
measure the volume of foods and drinking.
Based on Core ML and ARKit, the volume of foods and drinking is known, the weight,
calories and other nutrition information could be estimated. With calories and nutrition
information, users would able do nutrition management conveniently. And combine withother health management application, users could get more applicable nutrition management
suggestions to do health management. As a result, the healthy life expectancy and lifestyledisease problem would be solved.
Furthermore, to obtain more accurate result of calories and nutrition information, themodels of machine learning could be trained by developers themselves to meet specificrequirements. The new models would identify more categories and kinds of foods and drinks
to get more accurate nutrition value.

!60

Acknowledgements
First of all, thanks to Professor Tetsuro Ogi, who leaded me to do this research in these
two years. I would like to express the deepest appreciation to him, . Without his guidance andpersistent help this research, it would not possible to complete.
Thanks to Professor Naohiko Kohtake, who gave me helpful advices for my research,
particularly verification and validation of the application.
Thanks to Professor Tetsuya Toma, who supported in health management project.
Thanks to my parents, who sponsored me to finish my master studies and gave me achance to study in Japan and experience different culture.
Thanks to all lab members, who encouraged me and shared research experience with me.
This experience assisted me continue my research.
Thanks to all participants, who tried to use the application and answered the questionnaire
to help me finish the evaluation parts of the application.

July 2019
Yawen Sun

!61

Publication List
The following is a list of publications containing the work presented in this thesis as well
as some addition results.
1. AR Measurement Technology Applied to Nutritional Management, Asian Conferenceon Design and Digital Engineering, Yawen Sun, Tetsuro Ogi, 2018 November.
2. Nutrition Management Based on Deep Learning and AR Measurement technology,
Asian Conference on Design and Digital Engineering, Yawen Sun, Tetsuro Ogi, 2019
July.

!62

Reference
1.

“Healthy Life Expectancy: Key Points.” ScotPHO, www.scotpho.org.uk/populationdynamics/healthy-life-expectancy/key-points/.

2.

“Life Expectancy.” World Health Organization, World Health Organization, 6 July
2018, www.who.int/gho/mortality_burden_disease/life_tables/situation_trends/en/.

3.

“GHO | By Category | Healthy Life Expectancy (HALE) - Data by Country.” World
Health Organization, World Health Organization, apps.who.int/gho/data/
view.main.HALEXv?lang=en.

4.

“What Is Healthcare Management?” Healthcare Administration Degree Programs,
www.healthcare-administration-degree.net/faq/healthcare-management/.

5.

“60％以上の⼈が⽣活習慣病で死亡している！.” HelC＋（ヘルシー）,
www.health.ne.jp/library/detail?slug=hcl_3000_w3000680&doorSlug=life-style.

6.

Hoken, www.hoken-erabi.net/seihoshohin/kazunasemi/insu/medical/76.html.

7.

“Defining Health Promotion and Disease Prevention - RHIhub Toolkit.” Defining
Health Promotion and Disease Prevention - RHIhub Toolkit, www.ruralhealthinfo.org/
toolkits/health-promotion/1/definition.

8.

“What Is Health Promotion?” World Health Organization, World Health Organization, 3Aug. 2016, www.who.int/features/qa/health-promotion/en/.

9.

Health Japan 21, www.nibiohn.go.jp/eiken/kenkounippon21/en/.

10. “List of Targets.” Health Japan 21 (the Second Term) | Health Japan 21,
www.nibiohn.go.jp/eiken/kenkounippon21/en/kenkounippon21/mokuhyou.html.

!6311. “What Is MHealth (Mobile Health)? - Definition from WhatIs.com.” SearchHealthIT,
searchhealthit.techtarget.com/definition/mHealth.

12. MHealth: New Horizons for Health through Mobile Technologies. World HealthOrganization, 2011.

13. “What Is Health Apps? - Definition from WhatIs.com.” SearchHealthIT,
searchhealthit.techtarget.com/definition/health-apps.

14. “IOS - Health.” Apple (Malaysia), www.apple.com/my/ios/health/.

15. “ライフログテクノロジー株式会社 健康寿命を太く、⾧くする.” ライフログテ
クノロジー株式会社, calomeal.com/.

16. Krehl, W A. “The Role of Nutrition in Maintaining Health and Preventing
Disease.” Health Values, U.S. National Library of Medicine, 1983,
www.ncbi.nlm.nih.gov/pubmed/10260843.

17. “Pizza Calories.” CATEGORIES, www.calories.info/food/pizza.

18. “Food Lab inside Your Mobile Phone.” Fraunhofer Food Chain Management Alliance,
www.fcm.fraunhofer.de/en/beispiele11/food-lab-inside-your-mobile-phone.html.

19. “Food Lab inside Your Mobile Phone.” Fraunhofer Food Chain Management Alliance,
www.fcm.fraunhofer.de/en/beispiele11/food-lab-inside-your-mobile-phone.html.

20. “Pocket-Size Food Scanner.” Fraunhofer, www.fraunhofer.de/en/press/research-news/
2019/january/pocket-size-food-scanner.html.

21. Open Food Facts, world.openfoodfacts.org/discover.

!6422. Gigandet, Stephane. “Open Food Facts.” App Store, 3 Jan. 2013, apps.apple.com/nz/
app/open-food-facts/id588797948.

23. “What Is Machine Learning? A Definition.” Expert System, 5 Oct. 2017,www.expertsystem.com/machine-learning-definition/.

24. /@rajesh_khadka. “Machine Learning Types #2.” Medium, Towards Data Science, 8
Sept. 2017, towardsdatascience.com/machine-learning-types-2-c1291d4f04b1.

25. “Advantages and Disadvantages of Machine Learning Language.” DataFlair, 1 Mar.
2019, data-flair.training/blogs/advantages-and-disadvantages-of-machine-learning/.

26. “Core ML.” Core ML | Apple Developer Documentation, developer.apple.com/
documentation/coreml.

27. “Create ML.” Create ML | Apple Developer Documentation, developer.apple.com/
documentation/createml.

28. Sales. “Know the Augmented Reality Technology: How Does AR Work?” Cloud
Computing, www.newgenapps.com/blog/augmented-reality-technology-how-ar-works.

29. Apple Inc. “Augmented Reality.” Apple Developer, developer.apple.com/augmentedreality/.

30. Ashwini, Amit. “What Are the Advantages of Apple AR Kit ?” Digital Product Insights,
Digital Product Insights, 26 June 2017, www.cognitiveclouds.com/insights/what-isapple-ar-kit-and-what-does-it-do/.

31. “Nate Kaiser.” Nate Kaiser::Portfolio Website, njkaiser.github.io/portfolio/
compviz.html.

32. “Use the Measure App on Your IPhone, IPad, or IPod Touch.” Apple Support, 28 May
2019, support.apple.com/en-my/HT208924.
!6533. View Controller Programming Guide for IOS: Using Segues, 16 Sept. 2015,
developer.apple.com/library/archive/featuredarticles/ViewControllerPGforiPhoneOS/
UsingSegues.html.

34. “UIStoryboardSegue.” UIStoryboardSegue - UIKit | Apple Developer Documentation,
developer.apple.com/documentation/uikit/uistoryboardsegue.

35. “Pizza Calories.” CATEGORIES, www.calories.info/food/pizza.

36. “Creating an Image Classifier Model.” Creating an Image Classifier Model | Apple
Developer Documentation, developer.apple.com/documentation/createml/
creating_an_image_classifier_model.

37. Aizawa, Kiyoharu, and Makoto Ogawa. “FoodLog: Multimedia Tool for Healthcare
Applications.” IEEE MultiMedia, vol. 22, no. 2, 2015, pp. 4–8., doi:10.1109/mmul.
2015.39.

38. “Understanding World Tracking.” Understanding World Tracking | Apple Developer
Documentation, developer.apple.com/documentation/arkit/
understanding_world_tracking.

39. “Core ML Tutorial: Build an IOS App That Recognizes Hand-Drawn Digits.” Core ML- IBM Watson, 23 Feb. 2019, dataplatform.cloud.ibm.com/docs/content/wsj/analyzedata/ml-core-ml-mnist-tutorial.html.

40. Apple Inc. “Introducing ARKit 3 - WWDC 2019 - Videos.” Apple Developer,
developer.apple.com/videos/play/wwdc2019/604/.

41. Sekiguchi, Kentaro. “Museum Exhibition Linking with the Real World by Using
Augmented Reality Technology.” Keio Gijuku Daigaku Daigakuin Shisutemu Dezain
Manejimento Kenkyuka, Mar. 2014, pp. 101.

!66

42. Jordan, M. I., and T. M. Mitchell. “Machine Learning: Trends, Perspectives, andProspects.” Science, vol. 349, no. 6245, 2015, pp. 255–260., doi:10.1126/
science.aaa8415.

43. Kourou, Konstantina, and Themis P Exachors. “Machine Learning Applications inCancer Prognosis and Prediction.” Computational and Structural Biotechnology
Journal, vol. 13, 2015, pp. 8–17., doi:10.1016/j.csbj.2014.11.005.

44. Li, Bo, et al. “Acoustic Modeling for Google Home.” Interspeech 2017, 2017, doi:
10.21437/interspeech.2017-234.

45. Thakkar, Mohit. “Custom Core ML Models Using Create ML.” Beginning Machine
Learning in IOS, 2019, pp. 95–138., doi:10.1007/978-1-4842-4297-1_4.

46. Wu, Hsin-Kai, et al. “Current Status, Opportunities and Challenges of Augmented
Reality in Education.” Computers & Education, vol. 62, 2013, pp. 41–49., doi:10.1016/
j.compedu.2012.10.024.

!67

Appendix
Codes for Application
Core ML
import UIKit
import AVFoundation
import CoreML
import Visionclass CoreML: UIViewController, AVCaptureVideoDataOutputSampleBufferDelegate {@IBOutlet weak var cameraView: UIView!
@IBOutlet weak var button: UIButton!
@IBOutlet weak var classificationText: UILabel!

override func prepare(for segue: UIStoryboardSegue, sender: Any?) {let ViewController = segue.destination as! ViewController

if let text = classificationText.text {ViewController.text = text}}!68

private var requests = [VNRequest]()private lazy var cameraLayer: AVCaptureVideoPreviewLayer =AVCaptureVideoPreviewLayer(session: self.captureSession)
private lazy var captureSession: AVCaptureSession = {let session = AVCaptureSession()session.sessionPreset = AVCaptureSession.Preset.photoguard
let backCamera = AVCaptureDevice.default(for: .video),
let input = try? AVCaptureDeviceInput(device: backCamera)
else { return session }session.addInput(input)
return session}()

override func viewDidLoad() {super.viewDidLoad()
self.cameraView?.layer.addSublayer(self.cameraLayer)
let videoOutput = AVCaptureVideoDataOutput()videoOutput.setSampleBufferDelegate(self, queue: DispatchQueue(label: "MyQueue"))
self.captureSession.addOutput(videoOutput)
self.captureSession.startRunning()
setupVision()}!69

override func viewDidLayoutSubviews() {super.viewDidLayoutSubviews()
self.cameraLayer.frame = self.cameraView?.bounds ?? .zero}

func setupVision() {guard let visionModel = try? VNCoreMLModel(for: Inceptionv3().model)
else { fatalError("Can't load VisionML model") }let classificationRequest = VNCoreMLRequest(model: visionModel,
completionHandler: handleClassifications)classificationRequest.imageCropAndScaleOption = .centerCrop
self.requests = [classificationRequest]}

func handleClassifications(request: VNRequest, error: Error?) {guard let observations = request.results
else { print("no results: \(error!)"); return }let classifications = observations[0...4].compactMap({ $0 as? VNClassificationObservation }).filter({ $0.confidence > 0.3 }).sorted(by: { $0.confidence > $1.confidence }).map {(prediction: VNClassificationObservation) -> String in
return "\(round(prediction.confidence * 100 * 100)/100)%: \(prediction.identifier)"}!70

DispatchQueue.main.async {print(classifications.joined(separator: "###"))
self.classificationText.text = classifications.joined(separator: "\n")}}func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer:
CMSampleBuffer, from connection: AVCaptureConnection) {guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {return}var requestOptions:[VNImageOption : Any] = [:]if let cameraIntrinsicData = CMGetAttachment(sampleBuffer,kCMSampleBufferAttachmentKey_CameraIntrinsicMatrix, nil) {requestOptions = [.cameraIntrinsics:cameraIntrinsicData]}
let imageRequestHandler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer,orientation: CGImagePropertyOrientation(rawValue: 1)!, options: requestOptions)do {try imageRequestHandler.perform(self.requests)} catch {print(error)
}
}

!71

override func didReceiveMemoryWarning() {super.didReceiveMemoryWarning()
}

}

!72

ViewController
import UIKit
import SceneKit
import ARKit

final class ViewController: UIViewController {@IBOutlet weak var sceneView: ARSCNView!
@IBOutlet weak var targetImageView: UIImageView!
@IBOutlet weak var loadingView: UIActivityIndicatorView!
@IBOutlet weak var messageLabel: UILabel!
@IBOutlet weak var meterImageView: UIImageView!
@IBOutlet weak var resetImageView: UIImageView!
@IBOutlet weak var resetButton: UIButton!

@IBOutlet weak var newlabel2: UILabel!

fileprivate lazy var session = ARSession()
fileprivate lazy var sessionConfiguration = ARWorldTrackingConfiguration()
fileprivate lazy var isMeasuring = false;
fileprivate lazy var vectorZero = SCNVector3()
fileprivate lazy var startValue = SCNVector3()
fileprivate lazy var endValue = SCNVector3()
fileprivate lazy var lines: [Line] = []fileprivate var currentLine: Line?
!73

fileprivate lazy var unit: DistanceUnit = .centimetervar text:String?

@IBOutlet weak var label: UILabel!

override func viewDidLoad() {let str = text

if str!.contains("pizza"){label.text = "Please measure"}else {
label.text = "No information"}

super.viewDidLoad()
setupScene()}!74

override func viewWillAppear(_ animated: Bool) {super.viewWillAppear(animated)
UIApplication.shared.isIdleTimerDisabled = true}

override func viewWillDisappear(_ animated: Bool) {super.viewWillDisappear(animated)session.pause()}override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {resetValues()
isMeasuring = true
targetImageView.image = UIImage(named: "targetGreen")}

override func touchesEnded(_ touches: Set<UITouch>, with event: UIEvent?) {isMeasuring = false
targetImageView.image = UIImage(named: "targetWhite")
if let line = currentLine {lines.append(line)
currentLine = nil
resetButton.isHidden = false
resetImageView.isHidden = false
}
!75

}

override var prefersStatusBarHidden: Bool {return true}}

extension ViewController: ARSCNViewDelegate {func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) {DispatchQueue.main.async { [weak self] in
self?.detectObjects()}}

func session(_ session: ARSession, didFailWithError error: Error) {messageLabel.text = "Error occurred"}

func sessionWasInterrupted(_ session: ARSession) {messageLabel.text = "Interrupted"}

func sessionInterruptionEnded(_ session: ARSession) {messageLabel.text = "Interruption ended"
!76

}
}

extension ViewController {@IBAction func meterButtonTapped(button: UIButton) {let alertVC = UIAlertController(title: "Settings", message: "Please select distance unit
options", preferredStyle: .actionSheet)alertVC.addAction(UIAlertAction(title: DistanceUnit.centimeter.title, style: .default){ [weak self] _ in
self?.unit = .centimeter})
alertVC.addAction(UIAlertAction(title: DistanceUnit.inch.title, style: .default) { [weakself] _ in
self?.unit = .inch})
alertVC.addAction(UIAlertAction(title: DistanceUnit.meter.title, style: .default) { [weakself] _ in
self?.unit = .meter})
alertVC.addAction(UIAlertAction(title: "Cancel", style: .cancel, handler: nil))present(alertVC, animated: true, completion: nil)}

@IBAction func resetButtonTapped(button: UIButton) {!77

resetButton.isHidden = true
resetImageView.isHidden = true
for line in lines {line.removeFromParentNode()}lines.removeAll()}}

extension ViewController {fileprivate func setupScene() {targetImageView.isHidden = true
sceneView.delegate = self
sceneView.session = session
loadingView.startAnimating()
meterImageView.isHidden = truemessageLabel.text = "Detecting the world…"
resetButton.isHidden = true
resetImageView.isHidden = true
session.run(sessionConfiguration, options: [.resetTracking, .removeExistingAnchors])
resetValues()}

fileprivate func resetValues() {isMeasuring = false!78

startValue = SCNVector3()
endValue = SCNVector3()}

fileprivate func detectObjects() {guard let worldPosition = sceneView.realWorldVector(screenPosition: view.center) else
{ return }targetImageView.isHidden = false
meterImageView.isHidden = false
if lines.isEmpty {messageLabel.text = "Hold screen & move your phone…"}
loadingView.stopAnimating()
if isMeasuring {if startValue == vectorZero {startValue = worldPosition
currentLine = Line(sceneView: sceneView, startVector: startValue, unit: unit)}
endValue = worldPosition
currentLine?.update(to: endValue)
messageLabel.text = currentLine?.distance(to: endValue) ?? "Calculating…"

let newstring = messageLabel.text
let newnum = NSString (string : newstring!).doubleValue
let newweight1 = (newnum / 2) * (newnum / 2) * 1 * 1.05
!79let newweight2 = newweight1 * 2.66
let newweight3 = String(format: "%.2f", newweight1)
let newweight4 = String(format: "%.2f", newweight2)
newlabel2.text = "Weight \(newweight3) g, Calories \(newweight4) kCal" as String

}
}
}

!80

Line
import SceneKit
import ARKit

enum DistanceUnit {case centimetercase inchcase metervar fator: Float {switch self {case .centimeter:
return 100.0
case .inch:
return 39.3700787
case .meter:
return 1.0}}

var unit: String {switch self {case .centimeter:
return "cm"case .inch:!81

return "inch"case .meter:
return "m"}}var title: String {switch self {case .centimeter:
return "Centimeter"case .inch:
return "Inch"case .meter:
return "Meter"}}}

final class Line {fileprivate var color: UIColor = .white

fileprivate var startNode: SCNNode!
fileprivate var endNode: SCNNode!
fileprivate var text: SCNText!
fileprivate var textNode: SCNNode!
!82

fileprivate var lineNode: SCNNode?

fileprivate let sceneView: ARSCNView!
fileprivate let startVector: SCNVector3!
fileprivate let unit: DistanceUnit!

init(sceneView: ARSCNView, startVector: SCNVector3, unit: DistanceUnit) {self.sceneView = sceneView
self.startVector = startVector
self.unit = unit

let dot = SCNSphere(radius: 0.5)
dot.firstMaterial?.diffuse.contents = color
dot.firstMaterial?.lightingModel = .constant
dot.firstMaterial?.isDoubleSided = true
startNode = SCNNode(geometry: dot)
startNode.scale = SCNVector3(1/500.0, 1/500.0, 1/500.0)
startNode.position = startVector
sceneView.scene.rootNode.addChildNode(startNode)endNode = SCNNode(geometry: dot)endNode.scale = SCNVector3(1/500.0, 1/500.0, 1/500.0)

text = SCNText(string: "", extrusionDepth: 0.1)
text.font = .systemFont(ofSize: 5)!83text.firstMaterial?.diffuse.contents = color
text.alignmentMode = kCAAlignmentCentertext.truncationMode = kCATruncationMiddle
text.firstMaterial?.isDoubleSided = true

let textWrapperNode = SCNNode(geometry: text)
textWrapperNode.eulerAngles = SCNVector3Make(0, .pi, 0)
textWrapperNode.scale = SCNVector3(1/500.0, 1/500.0, 1/500.0)

textNode = SCNNode()
textNode.addChildNode(textWrapperNode)let constraint = SCNLookAtConstraint(target: sceneView.pointOfView)
constraint.isGimbalLockEnabled = true
textNode.constraints = [constraint]
sceneView.scene.rootNode.addChildNode(textNode)}func update(to vector: SCNVector3) {lineNode?.removeFromParentNode()
lineNode = startVector.line(to: vector, color: color)
sceneView.scene.rootNode.addChildNode(lineNode!)

text.string = distance(to: vector)
textNode.position = SCNVector3((startVector.x+vector.x)/2.0, (startVector.y+vector.y)/2.0, (startVector.z+vector.z)/2.0)!84

endNode.position = vector
if endNode.parent == nil {sceneView?.scene.rootNode.addChildNode(endNode)}}func distance(to vector: SCNVector3) -> String {return String(format: "%.2f%@", startVector.distance(from: vector) * unit.fator,
unit.unit)}

func removeFromParentNode() {startNode.removeFromParentNode()
lineNode?.removeFromParentNode()
endNode.removeFromParentNode()
textNode.removeFromParentNode()
}
}

!85

ARSCNView
import SceneKit
import ARKit

extension ARSCNView {func realWorldVector(screenPosition: CGPoint) -> SCNVector3? {let results = self.hitTest(screenPosition, types: [.featurePoint])
guard let result = results.first else { return nil }return SCNVector3.positionFromTransform(result.worldTransform)
}
}

!86

SCNVector3
import ARKit

extension SCNVector3 {static func positionFromTransform(_ transform: matrix_float4x4) -> SCNVector3 {return SCNVector3Make(transform.columns.3.x, transform.columns.3.y,transform.columns.3.z)}

func distance(from vector: SCNVector3) -> Float {let distanceX = self.x - vector.x
let distanceY = self.y - vector.y
let distanceZ = self.z - vector.z

return sqrtf((distanceX * distanceX) + (distanceY * distanceY) + (distanceZ *distanceZ))}

func line(to vector: SCNVector3, color: UIColor = .white) -> SCNNode {let indices: [Int32] = [0, 1]
let source = SCNGeometrySource(vertices: [self, vector])
let element = SCNGeometryElement(indices: indices, primitiveType: .line)
let geometry = SCNGeometry(sources: [source], elements: [element])
geometry.firstMaterial?.diffuse.contents = color
let node = SCNNode(geometry: geometry)!87return node}}
extension SCNVector3: Equatable {public static func ==(lhs: SCNVector3, rhs: SCNVector3) -> Bool {return (lhs.x == rhs.x) && (lhs.y == rhs.y) && (lhs.z == rhs.z)}}

AppDelegate
import UIKit@UIApplicationMain
class AppDelegate: UIResponder, UIApplicationDelegate {var window: UIWindow?
func application(_ application: UIApplication, didFinishLaunchingWithOptions
launchOptions: [UIApplication.LaunchOptionsKey: Any]?) -> Bool {return true}

func applicationWillResignActive(_ application: UIApplication) {}!88func applicationDidEnterBackground(_ application: UIApplication) {}func applicationWillEnterForeground(_ application: UIApplication) {}func applicationDidBecomeActive(_ application: UIApplication) {}func applicationWillTerminate(_ application: UIApplication) {
}
}

!89

Questionnaire!90