Internship Report
Improving estimation of grasping polygons
using gaze information
At Okayama University

Mr. Tanakan Pramot
Student ID 5810502334

Academic Year 2018
Department of Computer Engineering, Faculty ofEngineering
Kasetsart University

Abstract
The purpose of the report is to explain the details of writer’sinternship program at Okayama University, Japan in 2018. This
internship is part of the bachelor’s degree of computer engineering
program.
The topic of my project is “Improving estimation of grasping
polygons using gaze information”. I learned new methods to work on
research since I did not have any experience and this is my firstresearch project.
In this internship program, I gave 2 presentations, which are
mid-term and final presentations to show the progress and results of
my work to members of Monden laboratory which is my laboratory,
and Yamauchi laboratory.

Page (1) | 35

Acknowledgement
The internship report is the overall result of the Internshipprogram at Monden laboratory, Okayama University, Japan. Theinternship is part of the bachelor’s degree of computer engineering
program.
My internship project is improving estimation of grasping
polygons using gaze information. The ultimate purpose of my projectis to teach a robot how to grasp objects like a human does. For thispurpose, we first need to define graspable parts of the objects. To
define the graspable parts, I used human gaze information to improve
estimation of grasping polygons of an object.
I would like to thank the people who made this internshippossible. Asst. Prof. Zeynep Yucel, Prof. Akito Monden, Asst. Prof.
Pattara Leelaprute, Assoc. Prof. Arnon Rungsawang, Asst. Prof.
Bundit Manaskasemsak provided valuable assistance and and themembers of Monden laboratory supported my stay in Japan.
I hope this experiences and the knowledge that I gained from theinternship will improve my future work and be guidance to any young
students further.

Mr. Tanakan Pramot
Reporter
The last date of internship
31/7/2018

Page (2) | 35

Contents
Page
Abstract(1)
Acknowledgement(2)
Table of Contents(3)
List of Figures(4)
List of Tables(6)
Chapter 1 Introduction
1
Chapter 2 Background Knowledge and Related Work 3
Chapter 3 Methodology
5
Chapter 4 Results and Analysis
25
Chapter 5 Conclusion
27
Chapter 6 Problems and Comments
29
References
30
Appendix A Daily Reports
31
Appendix B Workplace Photos
36

Page (3) | 35

List of Figures
Page
Figure 1 Logo of Okayama University
1
Figure 2 Examples of Cornell University data set
3
Figure 3 Example of grasping polygons
4
Figure 4 Order of pictures in video clip
5
Figure 5 Experiment scenario
5
Figure 6 Gaze point on image
6
Figure 7 Discrete coordinates and Heat map
7
Figure 8 Heat map and Segmented gaze fixations
7
Figure 9 Before and after doing background subtraction 8
Figure 10 Before and after doing grayscale conversion 8
Figure 11 Before and after doing Hard-thresholding
9
Figure 12 Before and after applying Otsu’s adaptive filter
Figure 13 Grasping polygons in various objects
10
Figure 14 Example object in xy coordinates
11
Figure 15 Elliptic model
11
Figure 16 Overestimation from eigen value decomposition
Figure 17 Rotated binary image and its ellipse
12
Figure 18 Gaze blob13
Figure 19 Gaze’s morphological descriptor
14
Figure 20 Gaze’s positional descriptor
15
Figure 21 Polygon’s morphological descriptor
15
Figure 22 Polygon’s positional descriptor
16
Figure 23 Relation of So and Sp
17
Figure 24 Exponential model of So and Sp
17
Figure 25 Relation of Ro and Rp
18
Figure 26 Normal distribution model of Ro and Rp
18
Figure 27 Relation of θo and θp
19
Figure 28 Orientation difference plot
19
Figure 29 Von Mises distribution of θ
20
Figure 30 Relation of Ro and p
20
Figure 31 Rice distribution of Ro and p
21
Figure 32 Relation of Ro and p
21
Figure 33 Von Mises distribution of Ro and p
22
Figure 34 Relation of So and Sg
22

9

12

Page (4) | 35

Figure 35 Relation of Ro and Rg

23

Page 2 | 35

List of Figures
Figure 36 Relation of θFigure 37 Relation of Ro and g

Page
23
24

Page (5) | 35

List of Tables
Page
Table 1 Performance of gaze information
25
Table 2 Performance of heuristics
25
Table 3 Performance of blending gaze and heuristics 26

Chapter 1
Introduction
1.1 Motivations and Importance
Recently, robots can replace humans in many specialized tasks,
such as technical tasks in a factory. However, robots are still far fromdoing many tasks, which are very simple for humans. Therefore,
humans need to train robots more by using information fromthemselves to improve robot’s performance.

1.2 Objectives
1. Improve robot grasping performance by using human’s gazeinformation.
2. Teach robot to grasp like a human. (Future work)

1.3 Scope of Work
This outilne of the study canbe summarized as analysis of aground truth dataset, formulation of the problem in an analytical way,
software development on a computer platform and proof of
improvement in performance.

1.4 History and Detail of Company

Page (6) | 35

Figure 1 Logo of Okayama University
Name:
Okayama University
Description: The laboratoy is called ‘Software Measurement
and Analytics Laboratory’ ( a.k.a Monden Lab) and is part of
the Graduate School of Natural Science and Technology,
Okayama University. The main purpose of the laboratory isscientific research. Supervisor checks on research progress byhaving an appointment with researcher (or student) several times
in
a
week.
Position: Research student
Supervisor:
Prof. Akito Monden (Internship supervisor)
Asst. Prof. Zeynep Yucel (Project consultant)
Period: 1st June – 31st July 2018

1.5 Expected Benefits
1. Experience and knowledge about working as a researcher incomputer science field.
2. Learning about studying abroad in master’s degree andabroad life.
3. Get connections with foreign friends.

Page (6) | 35

Chapter 2
Background Knowledge and Related Work
2.1 Background Knowledge
In daily life, humans interact with various objects. From an earlyage, humans develop motor and cognitive skills to manage theirinteraction with objects. These possible actions that objects can be
employed for are called as "Affordance ".
Robots or computer programs cannot understand "Affordance "by default. Therefore, humans need to teach them by transferring
them their own knowledge. For this purpose, we use the "Attention"of humans, which they direct twords objects during their interaction
with them. "Attention" of humans can be determined from their
movement,
gaze,
or
brain
activities.
Beside knowing about “Affordance” and “Attention”, we needto carry out software coding to implement the solution.

2.2 Related Work
We use the freely available “Learning to grasp” data set ofCornell University, which contains 1034 images of a variety ofgraspable objects from various orientations.

Page (6) | 35

Figure 2 Examples of Cornell University data set
For each image, several grasping polygons are annotated byhuman coders. Grasping polygons are not always imagined for humangrasping, but they certainly do not exclude that either.

Figure 3 Example of grasping polygons

Page (6) | 35

Chapter 3
Methodology
3.1 Experiment3.1.1 Experiment video
A random subset of 432 images are selected, cropped, andresized to 1920 x 1080 resolution, to reduce clutter and provide aclear view. 12 video clips are built, displaying a series of 36 images
each (3 seconds for each image followed by a 2 seconds reset period).

Figure 4 Order of pictures in video clip
3.1.2 Subjects
Two subjects are instructed to image grasping the objects (i.e.
motivated) and the other two carry out a free viewing task (i.e.
unmotivated). Which mean motivated is watching for grasping andunmotivated is watching for curiosity.

Figure 5 Experiment scenario
Page (6) | 35

3.2 Gaze information
3.2.1 Gaze data
Gaze information is collected by an infrared sensor at afrequency of 70 Hz, and the gaze coordinates are logged together withcorresponding time stamps.

Figure 6 Gaze point on image
After we got gaze data we want to:- Get a continuous map (i.e. heat map) from discrete coordinates- Identify fixations (i.e. connected regions in the heat map)
- Segment out the fixations (i.e. blobs) for modeling gaze
3.2.2 Heat map
We build a continuous heat map from discrete coordinates byapplying a 2D Gaussian kernel at each gaze coordinate (gx, gy).

(k g ( x , y )=exp ⁡22− ( x − gx) +( y − g y )
2σ

2

)

Saliency of the screen coordinates (x,y) are assumed to be
independent, i.e. co-variance matrix is symmetric.
σ xy=σ yx =0

Page (6) | 35

Moreover, co-variance matrix is assumed to be isotropic (i.e.
spherical).
σ xx=σ yy =σAfter we compute kg(x,y) with support of 101x101 px with aproper σ (chosen using 3σ rule), we will get heat map from discretegaze coordinates.

Figure 7 Discrete coordinates and Heat map
3.2.3 Segmenting fixations
After we got heat map, we will segment out the gaze fixation(i.e. blobs) by binarize the heat map using Otsu’s adaptive
thresholding.

Figure 8 Heat map and Segmented gaze fixations
Now, we have gaze fixations that we can working on it insteadof using all discrete coordinates.

Page (6) | 35

3.3 Foreground estimation
3.3.1 Background subtraction
First of all, we will extract object from background image to get
the foreground image in three channels. This process is called as“Background subtraction”. We get RGB foreground image after doing
this process.

Figure 9 Before and after doing background subtraction
3.3.2 Grayscale conversion
We convert RGB image into grayscale image by taking a linear
combination of its channels.

Page (6) | 35

Figure 10 Before and after doing grayscale conversion

3.3.3 Hard-thresholding
Hard-thresholding of grayscale image does not always yield anice binarization. We need to apply some special method instead ofHard-thresholding.

Figure 11 Before and after doing Hard-thresholding
3.3.4 Otsu’s method
We apply Otsu’s adaptive filter to get a better binary image.

Page (6) | 35

Figure 12 Before and after applying Otsu’s adaptive filter
Now, we have segmented out the area of the object that we canworking on.

3.4 Grasping polygons
We have many objects in dataset, and all them are graspable sothere is at least one grasping polygon for each.
Despite we call them polygons, but they are actually rectangularregions. Two kinds of polygons are annotated, positive polygons(Conventional grasp) and negative polygons (Unconventional grasp).
In this research, we focus on positive polygons.

Page (6) | 35

Figure 13 Grasping polygons for various objects

3.5 Descriptors
We need to describe gaze, object and polygon properties torepresent their morphology and orientation. Descriptors are crucial in
exploring for the relations and correlations between grasping
intuitions, and gaze behavior and object properties.
3.5.1 Object descriptors
These descriptors help us understand about the shape andorientation of the object.
- Object’s size So
So is the first moment of the binary foreground image.
So =∑ ∑ F B ( x , y )xyFigure 14 Example object in xy coordinates
Page (6) | 35

- Orientation θo
We use an elliptic model to approximate θo. We use Eigen
decomposition of binary image (FB) to calibrate the elliptic model.
F B v p , s =λ p , s v p , s

Principal and secondary axes rp,s are aligned with eigen vectors of FB ,vp,s where p > s.
θo is the slope of the principal axisof the ellipse,
θo=arctan 2 ( v^ pv , v^ px )Where v^ px , y arecomponents of the unit vector v^ p

Figure 15 Elliptic model- Inertia ratio Ro
Ro is the ratio of secondary axis to the principal axis, i.e.
intuitively smaller eigen value to the larger one,
Ro=

λs
λp

But eigen value decomposition results in overestimation of ∨ ŕ p , s ∨Page (6) | 35

Figure 16 Overestimation from eigen value decomposition
Therefore, we rotate the image by -θo and compute the horizontal andvertical projections of this rotated image,
~x
~
~y
~
F B=∑ F B ( x , y ) F B =∑ F B ( x , y )yxThis can be used to approximate || rp,s || through,~x~y∨ ~r p ∨=∑ sgn ( F B ) ∨ ~r s ∨=∑ sgn ( F B )Then, R o= ~r s / ~r p and limited to the range (0,1].

Figure 17 Rotated binary image and its ellipse
3.5.2 Gaze descriptors
Gaze fixations are represented with blobs on the heat map which
can be treated in the same way as the object on binary image. (Elliptic
models) But this time we have two kinds of descriptors.

Page (6) | 35

Figure 18 Gaze blob
3.5.2.1 Morphological- Blob’s size Sg
Sg is the first moment of each blob on the fixation map.
(Similar to So.)- Orientation θg
θg is defined according to the elliptic model of the blob.
θ g =arctan 2 ( v^ gp / v^ gs )- Inertia ratio Rg
Rg is defined according to the elliptic model of the blob.
R g=~r gs / ~r gp

Page (6) | 35

Figure 19 Gaze’s morphological descriptor
3.5.2.2 Positional- Radial distance g
The radial distance is defined in a normalize manner as,δ g=∨ Co− Cg ∨So

Where Co,g are centroids of the object and gaze blob. This
normalization helps to make g independent of the area available.
- Angular position g
Angular position or g is defined as,ω g=arccos

(

⃗o− C
⃗g)
⃗r p ( C∨ r⃗ p ∨∨ ⃗C o− ⃗Cg ∨)Page (6) | 35

Figure 20 Gaze’s positional descriptor
3.5.3 Polygon descriptors
3.5.3.1 Morphological
Polygons have analogous morphological descriptors.
- Polygon’s size Sp
S p=wh- Orientation θp *(We are using θ instead of θ later)θ p =arctan 2 ( ⃗w / ⃗h )- Inertia ratio Rp
R p=h /w

Figure 21 Polygon’s morphological descriptor
Page (6) | 35

3.5.3.2 Positional
The positional descriptors are analogous as well.
- Radial distance p
δ p=∨C p − C o ∨So- Angular position pr⃗ p ( ⃗C p− ⃗C o)ω p=arccos(∨ r⃗ p ∨∨ ⃗Cp − ⃗Co ∨)Figure 22 Polygon’s positional descriptor

3.6 Empirical relations and models
After we have all of gaze, object, and polygon properties, we
expect the following pairs to have a certain degree of correlation:- So and Sp,g
- Ro and Rp,g
- θo and θp,g (θ)- Ro and p,g
- Ro and p,g

Page (6) | 35

3.6.1 Object size and polygon size: So and Sp

Figure 23 Relation of So and Sp
It is not surprising that grasping polygons of small objects are
small and the size grows in a somewhat negative exponential relation
with respect to the size of the object settling around a stable value forvery large objects.

Figure 24 Exponential model of So and Sp
We use an exponential model:f ( S p|C s , λ ) =C s ( 1− exp ⁡( − λ So ) )Where Cs is the scaling factor,  determines the rate of decay
and So stands for the size of the object.
Page (6) | 35

3.6.2 Object inertia ratio and polygon inertia ratio: Ro and Rp

Figure 25 Relation of Ro and Rp
There is no apparent relation between Ro and Rp. However, Rp
seems to be quite stable against Ro.

Figure 26 Normal distribution model of Ro and Rp
We assume Rp ~ N (rp, rp), where rp and rp are computed fromthe empirical observations.

Page (6) | 35

3.6.3 Object orientation and polygon orientation: θo and θp (θ)

Figure 27 Relation of θo and θp
We see a clear pattern that there is a certain offset between θoand θp. Thus we decide to examine θ = θo - θp.

Figure 28 Orientation difference plotθ suggests that principal axes of objects and polygons are often

perpendicular to each other. We decided to compute the cumulative
histogram and approximate it with Von Mises distribution.

Page (6) | 35

Figure 29 Von Mises distribution of θ
Von Mises distribution is the circular equivalent of the standard
normal distribution,f ( ∆ θ p ∨ μ∆ θ , κ ∆θ )=pp

exp ( κ ∆θ ( ∆ θ p − μ∆ θ ) )p

p

2 π I 0 ( κ∆ θ )pWhere θp and θp are analogous to the mean and standard
deviations of the normal distribution and I0 is the modified Bessel
function of order 0.
3.6.4 Object inertia ratio and polygon normalized radial distance:
Ro and p

Figure 30 Relation of Ro and p
Page (6) | 35

This curve indicates that p is larger for long objects and smaller
for round objects. For modeling this relation, we assume that the
projections of p on rp and rs come from a normal distribution with anonzero mean.

Figure 31 Rice distribution of Ro and pThen, p respects a Rice distribution, which is the magnitude of acircular bivariate normal random variable with potentially non-zero
mean.

(

f ( δ p ∨η , σ ) =C r τ +

δp
σ

2

exp

(

− ( δ 2p +η2 )2σ2) ( ))I0δη2
σ

Where Cr is a scaling factor,  is an offset value; and  and  are
the model parameters.
3.6.5 Object inertia ratio and polygon angular distance: Ro and p

Page (6) | 35

Figure 32 Relation of Ro and p
Objects with small Ro are expected to have a more pronounced
angular position than the ones with larger Ro. This expectation is
supported by the empirical findings. A Von Mises distribution with anadditional parameter for Ro would be the optimal. However, thecalibration process of such a model is likely to suffer from thesparsity of the space.

Figure 33 Von Mises distribution of Ro and p
Therefore, we propose binning the distribution in four equal
ranges of Ro and treating each of them with Von Mises distributions(of same parameters).
3.6.6 Object size and gaze size: So and Sg

Page (6) | 35

Figure 34 Relation of So and Sg
Motivated and unmotivated subjects present no significant
difference. Moreover, Sg does not seem to depend on S o to a large
extend. Therefore, we use a standard normal distribution.
S g N ( μsg , σ sg )3.6.7 Object inertia ratio and gaze inertia ration: Ro and Rg

Figure 35 Relation of Ro and Rg

Page (6) | 35

Motivated and unmotivated subjects present no significant
difference. Moreover, Rg does not seem to depend on Ro. Therefore,
we use a standard normal distribution.
R g N ( μsg , σ sg )3.6.8 Orientation difference: θFigure 36 Relation of θ
Unmotivated subjects present slightly better correlation withpolygon orientations. We use a Von Mises distribution model.

3.6.9 Object inertia ratio and gaze radial distance: Ro and g

Figure 37 Relation of Ro and gPage (6) | 35

Neither motivation nor Ro does not seem to have any effect on g

3.7 Estimation of grasping polygons
3.7.1 Strategies- We divide the set and apply 3-fold cross-validation.
- The training set is used to calibrate the models.
- For each image in the test set, one grasping polygon is
generated.
- Since the grasping polygon is defined by its descriptors, wedraw a random variable from the models of each descriptor.
- We use So and Ro of object for drawing random variables of Sp,
Rp, θp, p and p.
3.7.2 Evaluation metric
We use the conventional intersection over union (IoU) metricfor evaluating the performance. IoU is equivalent of Jaccard distancein the non-parametric form.
IoU =Mutual information
Joint entropy

In addition, we count the number of hits and misses. Misses are
the instance where the estimation does not have an intersection withany annotated polygon.

Page (6) | 35

Chapter 4
Results and Analysis
4.1 Results
4.1.1 Estimation performance using gaze information
Hit

Mi
ss
775 89Io
U
Gaze of motivated0.2
3
Gaze of782 82 0.3
unmotivated2
Table 1 Performance of gaze information
The gaze of unmotivated subjects yields better performance thanmotivated subjects in a consistent manner. Although this is asurprising finding, it was somewhat expected from the empirical
distributions.
4.1.2 Estimation performance using heuristics
Hit

Mi Io
ss U
Heuristic
688 176 0.2
4
Table 2 Performance of heuristic
Heuristics yield approximately same IoU as estimation frommotivated gaze, but the miss rate is much higher. This means thatwhen there is a hit, the IoU should be better thanmotivated gaze.
On the other hand, it performs much worse than unmotivatedgaze both in terms of IoU and hit rate.

Page (6) | 35

4.1.3 Estimation performance of blending gaze and heuristics
Hit

Mi
ss
775 89Io
U
Gaze of motivated0.2
3
Gaze of unmotivated782 82 0.3
2Gaze of motivated +864 0 0.2
heuristics
3
Gaze of unmotivated +864 0 0.3
heuristics
2
Table 3 Performance of blending gaze and heuristics
We estimate grasping polygons using gaze information. If thecentroid of the estimated polygon is not over the foreground, we
replace this estimation with heuristics.

4.2 Analysis
4.2.1 Using only gaze information or heuristics
As we can see on the Table 1, we expect that gaze of motivatedwill have better performance than gaze of unmotivated. However, we
got result that unmotivated have better performance than motivated.
The reason can be due to the active exploration of motivated subjects
for planning actions because when subjects imagine that they are
grasping object, they need to look around object to find efficient
graspable parts of objects. This make gaze of motivated are more
outside objects and give less performance than unmotivated.
Page (6) | 35

In the Table 2, we can see that we have many misses more thanusing gaze information. The reason is when coder annotated thegrasping polygons, some object has complicate shape. It makes some
centroid of grasping polygons are outside of the objects.
4.2.2 Using both gaze information and heuristics
After we combine using gaze information with heuristic in the
Table 3, we get more IoU than using only one of them. This isbecause when we use gaze information, it helps to define clearlygrasping part of the object. The reason is subject will looking mostly
in the foreground and same area in an object, so this will give better
random variable when we generate the polygon.

Page (6) | 35

Chapter 5
Conclusion
5.1 Work conclusion
The conclusion of this study is that gaze information can
improve grasping polygons and yield better evaluation performance.
This means that we can use our work as a baseline to teach robot tograsp like a human do as a future work.

5.2 Expectation
I expect that my internship project can be useful resource forresearchers working in robotics or artificial intelligence fields.

5.3 Benefits- Benefits to myself
I gain many experience from this internship such as experiencein working as a researcher, experience in studying abroad, andexperience in having foreign friends. I also get a chance to learning
new things and have a chance to consider about master’s degree incomputer science, Okayama university.
- Benefits to company
They have connection with me, and I think we have more strong
relationship between Kasetsart University and Okayama University.
They also learn some Thai culture in return as they teach me about
Japanese culture.
- Benefits to university
Similar to a company, university gets stronger relationship with
Okayama University and hopefully the two universities will havemore opportunities to have joint research projects in the future.

5.4 Swot Analysis- Strong point
I’m very good in communicating in English language. I caneasily communicate to people, who can understand English. I’m
friendly man so I met so many people from many nations. While I am
working, I’m working hard. While I am relaxing, I’m playing hard.
Page (6) | 35

- Weak point
I can’t communicate with Japanese language, so I missed many
opportunities to explore Japan by myself. I’m stubborn person
sometimes. When I stuck in some work, I will try my best to solve iteven it takes a long by not asking someone.
- Opportunity
I have many opportunities to learn new things. I also haveopportunity to meet many people from many nations like French,
Chinese, Vietnamese, Turkish, and Japanese.
- Obstacle
Language barrier between me and people who can’t talkEnglish. Sometimes I am lazy, so I miss opportunity to travel around
Japan because I decide to sleep in my dormitory than go out.

5.5 Impressive experience
All people I met are very nice. They are very kind and friendly.
People in my laboratory are very kind, friendly, and funny.
Especially, French guys. Not all people in my laboratory talk to me
because they don’t want to speak English but if I talk to them, they
will try to answer me kindly and slowly to make me understand. My
two sensei, Zeynep-sensei, and Monden-sensei are very kind too.
Monden-sensei is my internship supervisor, he will take care of me
about internship program. Zeynep-sensei is my project consultant, shehelps me a lot with my project and she also wants to a publish
research paper about my study. She will try her best to help me out inany circumstance even if I’m not a good researcher. I can tell that myimpressive experience is the people I met here.

Page (6) | 35

Chapter 6
Problems and Comments
Student
Problems
1. Language barrier.
2. Stubborn in problem.
Comments
1. Trying to learn Japanese.
2. Trying to ask for help.

Company
Problems
1. All project is large project.
2. Dormitory is expensive.
Comments
1. Need to adding some small project for short-term student
2. Trying to find a cheap one for student.

University
Problems
1. Too short period for attending internship program.
2. Too little scholarship.
Comments
1. Extend time for applying a program.
2. Give every student a scholarship.

Page (6) | 35

References

Page (6) | 35

Appendix A
Daily Reports
1st June 2018
Monden-sensei introduced me to Monden laboratory and show me mydesk. He tells me that we will talking about my project next week so
be prepare.

4th June 2018
Monden-sensei and Zeynep-sensei bring 3 projects to me and discuss
about them. They let me choose one topic that I am interested in. I
choose a topic about finding and improving graspable area of objects.
Zeynep-sensei is a supervisor of this project, so she gives me some
paper for understanding about the background of the project.

5th June 2018
After learning about the background of the project, she tells me about
experiment and tell me to prepare myself for experiment tomorrow.

6th June 2018
I need to do first half of experiment for gathering gaze data in theafternoon with the help of Huang Nguyen, French internship student.
I watch 6 videos with EEG headset on my head today. It really hurts.

7th June 2018
I need to do experiment again, today is final half of it. So, I watch 6
remaining videos then I finish gathering my data.

8th June 2018
After we finished experiment, Zeynep-sensei tells me that we will
working on code next week, so this week she wants me to write areport about background of the project for check my understanding
about what we are going to do next.

11th June 2018
Page (6) | 35

I hand over my report to Zeynep-sensei, she says its seems like I
understand about background of project now, so she gives me astandard code to work with and tells me to find IoU of the polygons ineach object.

12th June 2018
After I find IoU, Zeynep-sensei tells me that we want to know relation
between IoU and object size, so she tells me to do a backgroundsubtraction to find size of object. I start using matlib backgroundsubtraction

13th June 2018
After I use matlib background subtraction, I got RGB picture that
hard to compute object size. I change my method to opencv
background subtraction, it gives good results and we can now
compute object size.

14th June 2018
When we have object size, Zeynep-sensei tells me that we will findrelations between grasping polygons and object size, so she tells me
to find polygon’s size, distance between polygon centroid and centerof mass of the object, and orientation of polygons relative to theobject.

15th June 2018
I finished finding relations between grasping polygons and object
size, but the graphs of relations are hard to analyze and get a value to
use so I need to split object size into ranges (bins). Then I find meanvalue of each grasping polygon size respect to mean value of object
size in each bin and plot error bars using standard deviation of eachvalue.

18th June 2018
After I have mean value of each grasping polygon size, Zeynep-sensei
tells me that now we can randomly generate our own polygon using
normal distribution with mean value and standard deviation.
Page (6) | 35

19th June 2018
Working on generating random polygons.

20th June 2018
Working on generating random polygons. Changing range of binsfrom 200 to 500 because some of the bins don’t have any values,
when we use 200.

21st June 2018
Finish with generation of random polygons. Next, I need to checkperformance of our generation algorithm by using IoU compare to
ground-truth polygons.

22nd June 2018
I need to re-check the code because I get very low IoU values for our
randomly generated polygons incomparison to ground-truth
polygons.

25th June 2018
Working on re-code.

26th June 2018
After I re-code, I still get low IoU so Zeynep-sensei tells me thatmaybe it is because we using circular coordinates to randomly
generate the polygons. It gives quite different values compared to
ground-truth polygons. So, we need to change to an elliptic coordinate
frame that fits better to the object than circular model. But, now shetells me to count gaze inside and outside polygon then prepare my
mid-term project.

27th June 2018
I split gaze point into 2 group, gaze point of grasper and non-grasper.
In each group I split gaze point into 4 group as gaze in positive
polygons, in negative polygons, inside object (not in polygons), andoutside object (not in polygons). I can see that some gazes are outside
object but near the object, its mean sometimes people look at the edge
Page (6) | 35

of an object then Zeynep-sensei says that we need to contour object tocover that gaze points but now she tells me that I need to focus on my
mid-term presentation.

28th June 2018
Working on mid-term presentation.

29th June 2018
Working on mid-term presentation. Zeynep-sensei inform me about
mid-term presentation informations.

2nd July 2018
Before mid-term presentation, I need to present to Zeynep-sensei firstwith my French friends. So, my mid-term presentation goes well.
After presentation, my laboratory and yamauchi’s laboratory havepizza party together.

3rd July 2018
After mid-term presentation, I start working with Internship report.
Zeynep-sensei are busy about French student’s paper, so I need towork on my own in this week.

4th July 2018
Working on report and presentation to my Thai teacher.

5th July 2018
I am working on re-code some function and present my project to mythai teacher.

6th July 2018
Zeynep-sensei wants me to check about very small size and very big
size object with their foreground. And modify her function to makethat function works.
Page (6) | 35

9th July 2018
After I check on object and their foreground, I found that if object isvery small, some detail will be missing on its foreground. Otherwise,
if object is very big, some overuse detail will be appearing on its
foreground like shadow. And If object color is like background, thatpart of object will be missing on its foreground. I also finish
modifying her function.

10th July 2018
Zeynep-sensei tells me that we will start working with relation of
object, gaze, and grasping polygons properties after we finish finding
their properties from our elliptic model.

11th July 2018
Working on elliptic model.

12th July 2018
Zeynep-sensei ask me to try to generate grasping polygon by using
gaze information as a centroid and make grasping polygon
perpendicular to principal axis of elliptic model.

13th July 2018
After I finish generated grasping polygon, I evaluate performance ofgenerate grasping polygon.

17th July 2018
I see that elliptic model gives better performance than circular model.
This confirm that elliptic model is a right method for solve thisproblem.

18th July 2018
Now, we have object, gaze, and grasping polygons properties, weneed to find their relation and build a model for each relation.

19th July 2018
Page (6) | 35

Zeynep-sensei tells me that she will help me building a model foreach relation, so now she wants me to work on my final presentation
on next Monday.

20th July 2018
Working on final presentation.

23rd July 2018
Today is my final presentation day. My final presentation went well.
All thanks to Zeynep-sensei for all help.

24th July 2018
After a final presentation, I need to finish my Internship report then I
can give it to Zeynep-sensei to evaluate my report.

25th July 2018
Continue working on Internship report and Zeynep-sensei ask me todo some annotate data of my French friend.

26th July 2018
Continue working on Internship report and send an annotated data to
Zeynep-sensei.

27th July 2018
Continue working on Internship report.

30th July 2018
Finish working on Internship report and send to Zeynep-sensei forevaluation of my report.

31st July 2018
My last day at Okayama University and I finish all document that areneeded to be use in my Internship program.

Page (6) | 35

Appendix B
Workplace Photos
13 July 2018
Activities: Last day of my French friend, Hoang.

Page (6) | 35