Cooperative Education Report
Effect of grasping uniformity on estimation ofgrasping region from gaze data
At Software Measurement and Analytics
Laboratory of Okayama UniversityMs. Pimwalun Witchawanitchanun
Student ID 5910546686

Academic Year 2019
Department of Computer Engineering, Faculty ofEngineering
Kasetsart University

(1)Abstract
This report is a part of Cooperative Education at Software
Measurement and Analytics Laboratory of Okayama University,
which I spend time from June to July as an intern.
I was responsible for the maintenance of data of those whocome to experiment and bring data to analytics.
I was assigned to develop a graphical user interface in Java forannotation of several properties of the data set used in experiments
investigating the effect grasping motivation on gaze behavior.
I’ve have learned many things from this Cooperative Educationsuch as Data analysis, and how to use Python. In addition, I’ve
learned real work experience. This internship help me to be a
disciplined person and about responsibility towards the assignments.

(2)Acknowledgement
I had cooperative education opportunity
with Software
Measurement and Analytics Laboratory of Okayama University fromJune 2019 to July 2019. I am grateful for being given an internshipopportunity at Okayama University.
I would like to thank Kasetsart University for arranging thiscooperative education opportunity for me. I am also thankful for
Okayama University for accepting my cooperative education request.
I express my deepest thanks to Akito Moden, Professor ofOkayama University for arranged all facilities to make my life easier,and Zeynep Yucel, Assistant Professor of Okayama University, forgiving me necessary advice and helping me to solve the problem andtaking care of me so well.
Finally, I am very thankful to my family, my friends andmembers in Lab. for all their support during this cooperativeeducation period.
Ms. Pimwalun Witchawanitchanun
Reporter
The last date of co-op.......... / ............... / .............

(3)ContentsPageAbstract(1)
Acknowledgement(2)
Table of Contents(3)
List of Figures(4)
List of Tables(5)
Chapter 1 Introduction
1
Chapter 2 Background Knowledge and Related Work ?
Chapter 3 Methodology?
Chapter 4 Results and Analysis?
Chapter 5 Conclusion?
Chapter 6 Problems and Comments?
References?
Appendix A Daily Reports?
Appendix B Workplace Photos?

(4)List of Figures
Figure 1 ..........
Figure 2 ..........
Figure 3 ..........
Figure 4 ..........
Figure 5 ..........
Figure 6 ..........
Figure 7 ..........
Figure 8 ..........
Figure 9 ..........
Figure 10 ..........

Page?
?
?
?
?
?
?
?
?
?

(5)

List of Tables
Table 1 ..........
Table 2 ..........
Table 3 ..........
Table 4 ..........
Table 5 ..........
Table 6 ..........
Table 7 ..........
Table 8 ..........
Table 9 ..........
Table 10 ..........

Page?
?
?
?
?
?
?
?
?
?

1

Chapter 1
Introduction
1.1 Motivations and Importance
Humans-robot collaboration is a relatively new field. It would
be interesting if humans and robots could work together by interactingwith each other and doing manual tasks. Therefore, I used thiscooperative education opportunity for studying gaze behavior ofhumans over graspable objects, and evaluated estimation performanceof grasping regions from gaze data.

1.2 Objectives
1. To learn about data analysis.
2. To put programming skills in practice.
3. To apply knowledge from classroom to real work.
4. To prepare for the actual work in the future.

1.3 Scope of WorkIn this cooperative education, I was assigned to develop anapplication in Java for the purpose of annotating several properties ofgraspable objects.

1.4 History and Detail of Company
Software measurement and analytics laboratory at Okayama
University 1-1, Tsushima-Naka, 3-chome, Kita-Ku, Okayama, Japan.
The software measurement and analytics laboratory was started
in April 2015. It focuses on measurement and analysis of software
processes and products of large software development.

Organization and management model
Their research topics include software quality analysis, costestimation, project monitoring, human factors in softwaredevelopment, software security and protection.
Position or work detail
I was responsible for the maintenance of data of those whocome to experiment and bring data to analytics.
Name and position of student’s supervisor

2

Akito Monden as Professor.
Zeynep Yucel as WTT (Woman Tenure Truck) AssistantProfessor.
Internship/cooperative education period
From 3 June 2019 to 31 July 2019

1.5 Expected Benefits
I expected to get experience in data engineering and develop myprogramming skills.

3Chapter 2
Background Knowledge and Related Work
2.1 Background Knowledge- Java development skill
Java is the main programming language used in this cooperativeeducation. Java was first released by Sun Microsystems in 1995 andsince then it has been an important component in various applications
and websites. Java is used in laptops, game console, scientificsupercomputers, cell phones etc. and is considered to be fast, secure,and reliable.
- Human-robot collaboration
Human-robot collaboration is the study of collaborative
processes between humans and robotic agents, which work together toachieve shared goals. Numerous recent applications require robots towork alongside humans as capable members of human-robot teams.
Some applications include robots for homes, hospitals, and offices,space exploration and manufacturing.

2.2 Related Work- In literature, there exist various studies aimed at discovering
grasping affordances from gaze. Among those, several studies
investigate the effect of familiarity of objects and the intention of thesubjects (lifting or using) [1], readiness to act [2], as well as the effectof center bias on gazing and its evolution over time [3] anddistribution of gaze over functional and manipulative parts of “tools”[4]. The findings indicate that manipulative ends are gazed more often
and this effect becomes stronger, provided that the subject is familiar
with the object [1], is ready to act [2], and has the intention of “using”
rather than “lifting” [1].

4Chapter 3
Methodology
3.1 Visual stimuli and ground truth- DesignOverview
As visual stimuli, we used the freely available “Learning tograsp” data set of Cornell University, which contains 1034 images ofa variety of graspable objects from various orientations [5]. A random
subset of 432 images are selected from this set, cropped to a squareregion and resized to 1920×1080 resolution, to reduce clutter andprovide a clear view.

Figure 1: Example of the data set.
- Procedures
Two different coding tasks are performed on this subset as
follows.
Coding of grasping uniformity: Although all objects in data
set are graspable, we expect them to have either

5(i) A well-defined graspable part such as a handle (i.e. grasping
region is expected to be “uniform” across different graspers).
(ii) An intuitive region for grasping, even if they do not have awell-defined handle/grip (i.e. grasping region is expected to be“somewhat uniform” across different graspers.).
(iii) Neither a grip nor an intuitive grasping region (i.e. graspingregion is expected to vary largely across different graspers.).
We present these options to our coders and ask them to label
each image with one option or select another alternative as ‘I do notknow’.
As a result of this coding process, objects such as a mug,scissors are labeled as type-(i), whereas objects like bottle or
toothpaste are labeled as type-(ii), objects like shoes, vegetables arelabeled as type-(iii) .

Figure 2: User interface of grasping uniformity.

6

Flowchart of the annotation software for labeling grasping
uniformity
Program starts by receiving a string of inputs, composed of username and clip number. The clip number is used for get image filenames. These images are displayed one by one with the four options
alongside (see Figure 2.). For each image, user chooses a single
option and proceeds to the next image by pressing “Next” button. If
the user is not satisfied with his/her choice, he/she can press“Previous ” button and make a new selection. User cannot proceed,
unless he/she chooses one of the four options. Finally, the codings
are logged into a text file after the last image is annotated. After theprogram ends, it has 1 file in a directory with the same name as theuser.

7

Figure 3: Java program flowchart of grasping uniformity.
Coding of grasping regions: “Learning to grasp” data set
involves annotations of grasping regions on 2D image plane (defined
as quadrilaterals). However, since the data set is aimed at robot
grasping, there are some grasping regions in the ground truth, which
are unconventional for humans (e.g. open blades of scissors or tip of ascrewdriver, etc.). Therefore, we collected another set of ground truth
annotations regarding grasping region. Namely, a single grasping
region (imagining the grasper is a human) is labeled on each image ina similar way to original labeling (i.e. as a quadrilateral.)

8

Figure 4: Example of grasping regions.
Flowchart of the annotation software for labeling grasping
regions
Program starts by receiving a string of inputs composed of username and clip number. User marks 4 vertices on the image andproceeds to the next image by pressing “Next ” button. If the user is
not satisfied with the annotated region, he/she can press “Clear”
button and make a fresh start. User cannot proceed, if he/she marks
less than 4 vertices or does not define a simple quadrilateral. A set offour coordinates are saved into a text file. After the program ends, it
has 36 files under a directory with the same name as the user.

9- Tools
Eclipse: The Java IDE for Professional developer.
Draw.io: Program to drawing flowchart.

Example of Chapter 4
10

Chapter 4
Results and Analysis
4.1 Results

Table 1:

4.2 Analysis- Discuss and analyze the results

Example of Chapter 5
11

Chapter 5Conclusion- Conclude your work
- Expectation- Benefits
Benefits to yourself
Benefits to company
Benefits to university- Swot analysis (of yourself)- Impressive experience / special experience.......................................................................................................
................................................................................................................
................................................................................................................
................................................................................................................
...................

Example of Chapter 6
12

Chapter 6
Problems and Comments
Student
Problems
1. ............................................................................................................
...
2. ............................................................................................................
...
Comments
1. ............................................................................................................
...
2. ............................................................................................................
...

Company
Problems
1. ............................................................................................................
...
2. ............................................................................................................
...
Comments
1. ............................................................................................................
...
2. ............................................................................................................
...

UniversityProblems

13

1. ............................................................................................................
...
2. ............................................................................................................
...
Comments
1. ............................................................................................................
...
2. ............................................................................................................
...

Example of References
14References[1] Anna Belardinelli, Marissa Barabas, Marc Himmelbach,and Martin V Butz, “Anticipatory eye fixations reveal tool knowledge
for tool interaction”, Experimental Brain Research, vol. 234, no. 8,
pp. 2415-2431, 2016.
[2] Ettore Ambrosini and Marcello Costantini, “Body posturedifferentially impacts on visual attention towards tool, graspable, andnon-graspable objects.”, Journal of Experimental Psychology:
Human Perception and Performance, vol. 43, no. 2, pp. 360, 2017.
[3] Lotje Van Der Linden, Sebastiaan Mathôt, and FrançoiseVitu, “The role of object affordances and center of gravity in eyemovements toward isolated daily-life objects”, Journal of Vision,
vol. 15, no.5, pp. 8-8, 2015.
[4] N Natraj, YM Pella, AM Borghi, and LA Wheaton, “Thevisual encoding of tool--object affordances”, Neuroscience, vol. 310,
pp. 512-527, 2015.
[5] David Fischinger, Markus Vincze, and Yun Jiang,“Learning grasps for unknown objects in cluttered scenes”, Proc.
International Conference on Robotics and Automation, pp.609616, 2013.
[0] Java
8:Oracle,2018.
Available
https://docs.oracle.com/javase/8/docs/api/ [Accessed on: 29 July2019].
[0] Human-robot
collaboration:
2019.
Available
https://en.wikipedia.org/wiki/Human-robot_collaboration [Accessedon: 29 July 2019]

Example of Appendix
15Appendix A
Daily Reports
Date 3/6/2019
Read and try to understand paper of senior “Effect of motivation
on gaze behavior over time”

Date 4/6/2019
Find image in data set, find the amount of unique objects, findthe amount of grasping rectangle per image and find the amount ofgrasping rectangle per object.

16Appendix B
Workplace Photos
Date ....................................
Activities: ...............................................................................................
.