<figure><img src="image_1.png" /></figure>

HAL
open science

# Research on the Agricultural Skills Training Based on the Motion-Sensing Technology of the Leap Motion

Peng-Fei Zhao, Tian-En Chen, Wei Wang, Fang-Yi Chen

► To cite this version:

Peng-Fei Zhao, Tian-En Chen, Wei Wang, Fang-Yi Chen. Research on the Agricultural Skills Training Based on the Motion-Sensing Technology of the Leap Motion. 9th International Conference on Computer and Computing Technologies in Agriculture (CCTA), Sep 2015, Beijing, China. pp.277-286, 10.1007/978-3-319-48354-2_29. hal-01614219

HAL Id: hal-01614219

https://inria.hal.science/hal-01614219

Submitted on 10 Oct 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Research on the Agricultural Skills Training

## Based on the Motion-Sensing Technology of the Leap Motion

Peng-fei Zhao<sup>1,2,3,4,a</sup>, Tian-en Chen<sup>1,2,3,4,b,*</sup>, Wei Wang<sup>1,2,3,4,c</sup>, Fang-yi Chen<sup>1,2,3,4,d</sup>

<sup>1</sup>Beijing Research Center for Information Technology in Agriculture, Beijing 100097, China;

<sup>2</sup>National Engineering Research Center for Information Technology in Agriculture, Beijing 100097, China;<sup>3</sup>Key Laboratory of Agri-Infomatics, Ministry of Agriculture, Beijing 100097, China;

<sup>4</sup>Beijing Engineering Research Center of Agricultural Internet of Things, Beijing 100097, China;

<sup>a</sup>zhaopf@nercita.org.cn, <sup>b</sup>chente@nercita.org.cn,

<sup>c</sup>wangw@nercita.org.cn, <sup>d</sup>chenfy@nercita.org.cn

**Abstract:** With the increasing development of virtual reality technology, the motion-sensing technology used in agricultural skills training more widely, it plays a great role in promoting the agricultural production, scientific research and teaching. To break the space-time constraints, in order to train the farmers to know well the high precision agricultural skills, and increase the user's immersive and interactive, we propose a training method based on motion-sensing technology. This article in view of the grape vines binding technology, puts forward a kind of agricultural skills training methods based on the leap motion technology. Through maya bone modeling technology to realize 3-D simulation of grape vines, the system completes the interactive simulation of grape vines binding based on leap motion technology.

The experimental results show that the system can be a very good simulation of the grape vines binding process. System is stable, reliable and strong commonality, it can be used for simulating different plants vine binding, and the system innovative interactive, it can increases the user experience.

**Keywords:** Leap Motion, motion-sensing, agricultural, skills training

## 1 Introduction

Without using complex control equipment, the motion-sensing technology allow people use the body movements and digital equipment interact with the environment, and in accordance with people's movements to complete a variety of commands. Motion-sensing technology used in agricultural skills training, can solve many problems, for example, the experimental equipment that is insufficient, long time consuming problem. So, the technology can improve teaching efficiency and quality[1]. Training system based on the motion-sensing technology can
demonstrate farming activities of seeding simulation, crops such as fruit picking, crop cultivation
simulation. This article presents an interactive design training method for virtual agriculture grape
park based leap motion motion-sensing technology. First, build a three-dimensional grapes park
foundation template library and grape dimensional model, and then track user gestures, and
identify the user crawling vines, enhance vines, binding vines gestures, etc. When a series of
binding vine action is completed, by setting a good UI interface, the user is prompted to re-tie the
vines or success to bind vines.

## 2 Virtual 3D model

Traditional agricultural skills training let the farmers learn the technology in indoor classroom, then the farmers go to work in the filed, the process is often greatly limited by the space-time, weather factors, material, etc. Using the virtual farm training, the farmers can avoid damaging the crops because of mistake of farming. In agricultural training using a virtual farms, farmers can change the farmland environment and cultivation measures, directly observe changes in crop growth and yield, quality, so as to deepen understanding of the new technology, new varieties and master degree. The traditional agricultural science and technology promotion mode can not achieve this effect.

Virtual farm plant modeling refers to the use of physical or mathematical method, the mathematical model obtained to describe the system needs to simulation, it is essential for the digital simulation steps[2]. Three-dimensional modeling technology is mainly divided into computer modeling and software modeling. Due to computer modeling is composed of surface, the technology require higher performance computer, and many difficulties exist in the process of model building. So, the system adopts the Maya software modeling.

### 2.1 Maya Modeling Technology

Maya software is a leader in the field of three-dimensional digital animation and visual effects. The Maya function is flexible operation, highly realistic rendering, and maya integrates the animation with digital effects technology. Moreover, it integrates 3D modeling, animation, special effects with the rendering plate that provides a variety of tools for making model and animation need[3]. From the actual physical simulation to character animation, and the particle system complex, maya can almost making all of CG(Computer Graphics) forms.

Types of modeling in Maya software is divided into NURBS modeling, Polygons modeling, Subdivision modeling. The establishment of bone is the process of using joint and bone building level connection structure. The bones are the joint structural level, it can be animated and positioned the deformable objects. The user can make objects become joints and bone sub objects, and use the skeleton to control the movement of objects.

### 2.2 Grapes vines modeling

In this system, according to the laws of the grape phyllotaxis, we build the model of grape leaves through the Maya software. In this method, through key nodes and nurbs curve, we build the structure of branches. [4]. The construction of the grapes 3D model steps are as follows:

First, through the digital imaging equipment to capture static grape appearance, and the photo
processing for texture mapping real.

Second, consult the grape vine growth morphology, in accordance with the law, construct three-dimensional grape contour model.

Third, according to the laws of the grape phyllotaxis, build a virtual 3 d model of grape leaves.

Fourth, according to the grape inflorescence rule and the growth cycle of the fruit, build relevant three-dimensional form in different periods the grapes.

Fifth, in the virtual model of grape vines, to join the bone invasion, simulate the grapes and vine deformation.

As shown in Fig.1, Fig.1-a according to the growth rule of nature, extract the grape contour skeleton. Fig.1-b through the optimization design of pipelines, the creation of fruit and three dimensional profile. Fig.1-c through texture mapping and the optimized model, 3D virtual simulation of grape.

<figure><img src="image_3.png" /><figcaption>Fig.1. The construction of the grapes 3D model</figcaption></figure>

# 3 Motion-sensing technology

## 3.1 Kinect motion-sensing Technology

Motion-sensing interaction devices provide the user with high accuracy and low cost requirements, the user can easily distinguish different gestures, and the user can define a gesture to grasp virtual objects, or make a pull or folding finger movements.

Kinect is the use of peripheral body sense device most widely, it can complete real-time capture and tracking of human body in real-time. But kinect recognition accuracy is about 4mm, which requires the user need to have larger movement range[5]. And 30 frames per second capture ability make equipment cannot distinguish the rapid movements of players, delay is very bad. Because the weakness of precision and delay temporarily, the kinect is unable to solve on the wrist precision farming activity.
## 3.2 Leap Motion motion-sensing Technology

The sensing orange of Leap Motion accuracy can reach one percent mm, so it can let a person directly control computer through the fingers, including image scaling, moving, rotating and instruction operation, precise control, etc[6]. In this space that your 10 finger movements can be tracking in real-time, error in the 1/100 mm. The leap motion is superior to Kinect in the interactive speed and accuracy that can guarantee the user successfully completed more accurate, more precise farming activities. As shown in Fig.2, Leap motion captured five fingers outstretched palm movement [7].

<figure><img src="image_4.png" /><figcaption>Fig.2. Hand simulation</figcaption></figure>

## 3.3 Custom gesture model

Through the Maya modeling software, we can creat the 3-D hands model, as shown in Fig.3. In the system, the model is imported into the system, and we create the respond events of the hands[8]. In response to an event trigger, the system can complete the simulation of gestures.

<figure><img src="image_5.png" /><figcaption>Fig. 3. Hands gesture model</figcaption></figure>

As shown in Fig.4, Fig.4-a, in the real world, the system capture the left hand stretched out, and in the system, the response effect of virtual left hand.As shown in Fig.4-b, in the system, the effect of virtual left hand clenched.
<figure><img src="image_6.png" /><figcaption>Fig.4. Left hand status</figcaption></figure>

As shown in Fig.5, Fig.5-a in the real world, the system capture the right hand stretched out, and in the system, the response effect of virtual right hand.As shown in Fig.5-b, in the system, the effect of virtual right hand clenched.

<figure><img src="image_7.png" /><figcaption>Fig.5. Right hand status</figcaption></figure>

3.4 Inverse Kinematics

Inverse dynamics is a method to calculate the root nodes information through the modeling nodes information. In a nutshell, IK (Inverse Kinematics) algorithm confirms the location information of end, according to the position information of the end nodes derivate the position information of skeletal class N chain of the father[9].Through this method specifies the role of "position", namely the first rendered roles into bone joint structure, the so-called position expressed as the angle formed therefrom, including the state vector θ and the direction vector and the position of the node. For the IK algorithm, the basic formula is:

$$ \theta = f^{-1}(X) $$
Among them, X is usually the end location information, and use this algorithm to calculate to meet the conditions of the $\theta[10]$. As shown in Fig.6, we add the joint points in the vine branches and use IK inverse dynamics, the users can hold the vines users and the vines around the key points for spatial arbitrary direction of movement.

<figure><img src="image_8.png" /><figcaption>Fig.6. Joint point graphic</figcaption></figure>

## 3.5 System working process

After the user login, the system will automatically peripheral initialization, and check whether the normal peripheral is not connected or the driver is not installed, then the system prompt error and exit the training system. If the connection is normal, the system will pop up a simple introduction to the gesture operation, as shown in Fig.7. Next, the user hands could be detected in the Leap Motion space, with the left hand grabbing vine, right hand to pick up tying vine machine, and then the vines binding in the right position. The system will be based on the vine binding position to judge whether the user is tied to vines success. Tied cane is successful, then the system exit; If bind cane not successful, the system will prompt the user for training again.

System work process is shown in figure.

<figure><img src="image_9.png" /><figcaption>Fig.7. Operation method</figcaption></figure>
## 4 The experimental results

User login, such as the system peripherals are connected properly, then start tying the rattan movement simulation. As shown in Fig.8, the user's left hand appears in the Leap Motion detection range, there will be left hand in the virtual system, and prompts the user use the left hand grab the vine. As shown in Fig.9, users grab the vines, according to the arrow tips and improve near to a grape vines upward.

<figure><img src="image_10.png" /><figcaption>Fig.8. Virtual left hand</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Fig.9. Grasp the vine</figcaption></figure>

After users use the left hand complete a series of actions and keep the state invariant. When users right hand appear in the Leap Motion range of visibility, the system appears virtual right hand with tied rattan machine. At a specific location, the vines machine will be fixed on the grape vine, as shown in Fig.10. When bound nodes meet the requirements, system prompts tied rattan successfully, and exit the system, as shown in Fig.11.

<figure><img src="image_12.png" /><figcaption>Fig.10. Hand tied the vine</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Fig.11. Complete grasp vine</figcaption></figure>

As shown in Fig.12, it is the system sketch. The frame rate of the system is 54 frames / sec. As shown in Fig.13, it is the sketch that how to operate the system.
<figure><img src="image_14.png" /><figcaption>Fig.12. System sketch</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>Fig.13. Operate the system</figcaption></figure>

According to the above results, compared with the reference[5], Leap Motion can capture users movements, such as, finger movements. Moreover, the system runs smoothly and the FPS can reach 54 frames /sec, so that the system consumes less IT resources. Finally, the cost of the system hardware is low, and the operation is simple, the method is suitable for all kinds of agricultural training. So, it is feasible to carry on the agricultural training that using Leap Motion to capture the finger movements.

## 5 Conclusions

Compared with the teaching system of motion-sensing technology based on current, the system aimed at the agricultural field demand, the method can meet high precision farming gesture operation, and achieve fast response, low delay capture effect. Through the teaching system, whenever and wherever, the farmers can be better familiar with the technology of tying vines, and break the limit of space-time to grasp the essentials of tying vines. Achieve the demand of science and technology, science and technology education peasants farming, realize the modernization of agriculture.

The system can make full use of human gestures convenience, make the interactive design and experience of agricultural park is more natural, more real, and can meet the agriculture park aided design, virtual experience farming operations and science education application requirements.

## Acknowledgment

Funds for this research was provided by the Technology Innovation Ability Construction Projects of Beijing Academy of Agriculture and Forestry Sciences(KJCX20140416),and The Beijing Municipal Science and Technology Projects (D141100004914003). The research work also has been done under the help of the team of the Information engineering.
References

1. Xu Meng, Sun Shou-qian, Pan Yun-he. The Research of Motion Control in Virtual Human[J]. Journal of system simulation, 2003, 15 (3): 338-340.

2. Li Hai-lin, Luo Chang-yuan, Tang Jun. Application of virtual agriculture technology[J].Computer Engineering and Design, 2008, 8 (29): 2059-2061.

3. Qiu Ya-ping. Design of Characters Walking Movement Based on Maya Technology[J]. Computer and Modernization. 2010, 3 (4): 22-25.

4. Liu Chong-huai, Feng Jian-can, Jiang Jian-fu. Cluster Analysis of Chinese Wild Grape Species Based on Morphological Characters[J]. Journal of Plant Genetic Resources, 2011, 2 (6): 847-852.

5. Fan Jing-chao, Zhou Guo-min. Kinect based Agricultural virtual teaching[J]. Journal of Anhui Agricultural Sciences, 2014, 42 (12): 3706-3709.

6. LeapMotion SDK: https://developer.leapmotion.com

7. Leap Motion Product: https://www.leapmotion.com

8. Chen Hong, Ma Qin, Zhu De-hai. Research of Interactive Virtual Agriculture Simulation Platform Based on Unity3d[J]. Journal of agricultural mechanization research, 2012, 34(3):181-186.

9. Li Zhen-long, Peng Ya-xiong. The research based on the 3D inverse dynamics of IK algorithm[J]. Image Processing and Multimedia Technology, 2013,32 (24): 33-36.

10. Tolani D, Goswanmi A, Balder NI. Real time inverse kinematics techniques for anthropomorphic limbs[J]. Graphical Models, 2000, 62(5): 353 - 358.