Robot controlling method using portable device including touchscreen

A method for controlling a robot device using a portable terminal including a touchscreen is provided. The method includes displaying an enable button in a first area of the touchscreen, displaying an emergency stop button in a second area of the touchscreen, displaying a plurality of robot control buttons in a third area of the touchscreen, in response to simultaneously receiving from a user an input on the first area and an input on the third area, transmitting a robot control signal to a control device configured to control the robot device, and in response to receiving an input from the user on the second area, transmitting an emergency signal to the control device.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority under 35 U.S.C § 119 to Korean Patent Application No. 10-2019-0164207 filed on Dec. 10, 2019 in the Korean Intellectual Property Office, the entire contents of which are hereby incorporated by reference.

TECHNICAL FIELD

The present disclosure relates to a method for controlling a robot using a portable terminal including a touchscreen, and more specifically, to providing a user interface for robot controlling using a portable terminal.

BACKGROUND ART

The large-scale manufacturing process, in-line manufacturing, and increased automation facilities have led into the rapid increase in the use of industrial robots at the industrial sites. As a method for training robots at the industrial sites, a method such as direct training or training using teaching pendant is used. Meanwhile, recently, in addition to a dedicated pendant, a method for controlling the operation of a robot using various control devices such as smartphones, tablet PCs, joysticks, keyboards and so on has been introduced.

In the case of a dedicated pendant, it is implemented in consideration of the safety of user such that the user is able to control the robot only by pressing an operation button while pressing the enable button that is implemented as a physical button at the same time. On the other hand, many robot control methods using tablet PCs or the like developed recently are designed without considering the safety of the users. In this case, the user can be placed in a dangerous situation by a robot control command input contrary to the user's intention, or the user may not able to be fast enough to stop the robot in an emergency situation, and a safety accident can occur.

SUMMARY

In order to solve the problems described above, the present disclosure provides a method for controlling a robot device.

There is provided a method for controlling a robot device, with which it is possible to use a user interface optimized for each user, since a user can adjust positions, sizes, and shapes of an enable button and an emergency stop button according to his or her preference at a user terminal.

In addition, there is provided a method for controlling a robot device, which is capable of quickly responding to an emergency situation by, in response to detecting a registered posture of the user terminal, impact and shaking of the user terminal, a user gesture, a voice command, and the like by an accelerometer, an image sensor, a sound sensor, and the like of the user terminal, transmitting an emergency signal for stopping the operation of the robot device.

The present disclosure may be implemented in various ways, including a computer readable storage medium that stores a method, a system, an apparatus, or instructions.

The method for controlling a robot device using a portable terminal including a touchscreen according to an exemplary embodiment may include displaying an enable button in a first area of the touchscreen, displaying an emergency stop button in a second area of the touchscreen, displaying a plurality of robot control buttons in a third area of the touchscreen, in response to simultaneously receiving from a user an input on the first area and an input on the third area, transmitting a robot control signal to a control device configured to control the robot device, and in response to receiving an input from the user on the second area, transmitting an emergency signal to the control device. In this case, the emergency signal may be a signal for stopping operation of the robot device.

According to an embodiment, the third area may be located in a central area of the touchscreen, and the first area and the second area may be located in a border area surrounding the central area.

According to an embodiment, in a right-handed mode of the portable terminal, the enable button may be displayed on a left side of the touchscreen, and the emergency stop button may be displayed on a right side of the touchscreen. In addition, in a left-handed mode of the portable terminal, the enable button may be displayed on the right side of the touchscreen, and the emergency stop button may be displayed on the left side of the touchscreen.

According to an embodiment, a method for controlling a robot device may further include displaying a plurality of guide blocks on the border area, receiving, from the user, a first user input for selecting at least one of the plurality of guide blocks, displaying the enable button in the first area based on the first user input, receiving, from the user, a second user input for selecting at least one of the plurality of guide blocks, and displaying the emergency stop button in the second area based on the second user input.

According to an embodiment, the size of the first area for displaying the enable button is changeable by the user, and the size of the second area for displaying the emergency stop button is changeable by the user.

According to an embodiment, the shape of the first area for displaying the enable button is changeable by the user, and the shape of the second area for displaying the emergency stop button is changeable by the user.

According to an embodiment, a method for controlling a robot device may further include, in response to detecting, by an accelerometer of the portable terminal, a fall, or a shaking of the portable terminal of a predetermined threshold value or above, or a predetermined posture, transmitting an emergency signal to the control device.

According to an embodiment, the method for controlling a robot device may further include, in response to detecting a predetermined gesture by an image sensor of the portable terminal, transmitting an emergency signal to the control device.

According to an embodiment, the method for controlling a robot device may further include, in response to detecting a predetermined voice command by a sound sensor of the portable terminal, transmitting an emergency signal to the control device.

According to an embodiment, the portable terminal may be a general-purpose tablet computer.

DETAILED DESCRIPTION OF THE INVENTION

Hereinafter, specific details for the practice of the present disclosure will be described in detail with reference to the accompanying drawings. However, in the following description, detailed descriptions of well-known functions or configurations will be omitted when it may make the subject matter of the present disclosure rather unclear.

In the accompanying drawings, the same or corresponding components are given the same reference numerals. In addition, in the following description of the embodiments, duplicate descriptions of the same or corresponding components may be omitted. However, even if descriptions of components are omitted, it is not intended that such components are not included in any embodiment.

Advantages and features of the disclosed embodiments and methods of accomplishing the same will be apparent by referring to embodiments described below in connection with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below, and may be implemented in various different forms, and the embodiments are merely provided to make the present disclosure complete, and to fully disclose the scope of the invention to those skilled in the art to which the present disclosure pertains.

The terms used herein will be briefly described prior to describing the disclosed embodiments in detail.

The terms used herein have been selected as general terms which are widely used at present in consideration of the functions of the present disclosure, and this may be altered according to the intent of an operator skilled in the art, conventional practice, or introduction of new technology. In addition, in a specific case, a term is arbitrarily selected by the applicant, and the meaning of the term will be described in detail in a corresponding description of the embodiments. Therefore the terms used in the present disclosure should be defined based on the meaning of the terms and the overall contents of the present disclosure rather than a simple name of each of the terms.

As used herein, the singular forms ‘a,’ ‘an,’ and ‘the’ are intended to include the plural forms as well, unless the context clearly indicates the singular forms. Further, the plural forms are intended to include the singular forms as well, unless the context clearly indicates the plural forms.

Further, throughout the description, when a portion is stated as “comprising (including)” a component, it intends to mean that the portion may additionally comprise (or include or have) another component, rather than excluding the same, unless specified to the contrary.

FIG. 1is a diagram illustrating a robot device controlling system100using a portable terminal including a touchscreen according to an exemplary embodiment. As illustrated, the portable terminal110may be configured to communicate with a control device120through a plurality of first to third communication connections140,150, and160. The control device120may be configured to communicate with a robot device130through a fourth communication connection170.

According to the present disclosure, the portable terminal110may be any device that includes a touchscreen capable of outputting data to a user and receiving a touch input from the user, and a plurality of communication modules capable of communicating with external devices. For example, the portable terminal110may be a general-purpose tablet computer, a smart phone, a laptop computer, and the like. The user may input a command for controlling the robot device130using the portable terminal110. The command input by the user to the portable terminal110may be transmitted to the robot device130via the control device120.

In an embodiment, an enable button112and an emergency stop button114may be displayed on the touchscreen of the portable terminal110. In the present disclosure, the enable button112refers to a button for transmitting a command for controlling the robot device130to the control device120, and when the user touches the robot control button (not shown) while touching on the enable button112, a robot control command corresponding to the robot control button may be transmitted from the portable terminal110to the robot device130via the control device120. That is, while receiving the touch input on the enable button112area from the user, the portable terminal110may transmit the robot control command to the control device120in response to receiving an input on the robot control button area. On the other hand, the portable terminal110may not transmit the robot control command to the control device120when receiving only the input on the robot control button area.

In the present disclosure, the emergency stop button114is a button for stopping all operations of the robot device130, and when the user touches the emergency stop button114, the portable terminal110may transmit an emergency signal to the control device120, and the control device120may transmit an emergency stop command to the robot device130. That is, the portable terminal110may transmit the emergency signal to the control device120in response to receiving a touch input on the emergency stop button114area from the user. Unlike the other robot control buttons, the user is able to stop all operations of the robot device130by simply touching the emergency stop button114without having to touch the enable button112together, and this is for the user to easily and quickly stop the robot device130in an emergency situation.

The robot control button may be displayed in an area on the touchscreen where the enable button112and the emergency stop button114are not displayed, and a plurality of buttons may be displayed for precise control of the robot device130. For example, the robot control button may include a jog dial, a mode change switch, a motor key, a lock key, a robot selection key, an operation mode key, a speed key, a cursor key, a stop key, an OK key, and the like.

As illustrated, in a right-handed mode of the portable terminal110, the enable button112may be displayed on the left side of the touchscreen, and the emergency stop button114may be displayed on the right side. In this case, a user who generally uses the right hand may hold the portable terminal110in the left hand or both hands, and touch the robot control button with the right hand while simultaneously touching the enable button112with the left hand (e.g., with the thumb of the left hand). To this end, the enable button112may be placed on the left side of the lower end of the touchscreen.

In addition, the emergency stop button114may be placed on the right side such that, in the event of an emergency, the user is easily able to touch it with the right hand, which is the hand the user generally uses. In this case, a vertical length of the emergency stop button114may be greater than the vertical length of the enable button112. For example, the vertical length of the emergency stop button114may be set to be the same as the vertical length of the touchscreen. Additionally or alternatively, a horizontal length of the emergency stop button114may be greater than the horizontal length of the enable button112. InFIG. 1, the portable terminal110is displayed in a portrait mode, but not limited thereto, and when the portable terminal110is displayed in a landscape mode, the enable button112and the emergency stop button114may be rearranged to be displayed on the left and right sides of the touchscreen, respectively, while the user is holding the portable terminal110.

A command input by the user through the touchscreen of the portable terminal110may be transmitted to the control device120through the first to third communication connections140,150, and160. The first, second, and third communication connections140,150, and160between the portable terminal110and the control device120may be wired or wireless communication connections. For example, each of the first, second, and third communication connections140,150, and160may be a USB connection, a WiFi® connection, and a Bluetooth® connection. In addition, various communication methods such as Ethernet connection, mobile communication connection, ZigBee® connection, and so on may be used. In the figures, three communication connections are illustrated between the portable terminal110and the control device120, but the present disclosure is not limited thereto, and more than three communication connections may be provided.

The control device120configured to control the robot device130, and, in response to receiving a robot control command from the portable terminal110, may transmit a corresponding command to the control device120. In an embodiment, the control device120may include a plurality of communication modules for communicating with the portable terminal110and the robot device130, a controller for generating a robot control signal for controlling the operation of the robot device130, and the like. The fourth communication connection170between the control device120and the robot device130may be a wired or wireless communication connection. In an embodiment, the fourth communication connection170may be a wired communication connection for stable control of the robot device130and prevention of safety accidents. For example, the fourth communication connection170may be a USB connection, an Ethernet connection, and the like.

FIG. 2is a diagram illustrating a left-handed mode of the portable terminal110according to an exemplary embodiment. In an embodiment, the user may select one of a right-handed mode or a left-handed mode of the portable terminal110. When the user selects the left-handed mode, an enable button212may be displayed on the right side of the touchscreen, and an emergency stop button214may be displayed on the left side.

A user who generally uses the left hand may hold the portable terminal110in the right hand or both hands, and touch the robot control button (not illustrated) with the right hand while simultaneously touching the enable button212with the right hand (e.g., with the thumb of the right hand). To this end, the enable button212may be placed on the right side of the lower end of the touchscreen.

In addition, the emergency stop button214may be placed on the left side such that, in the event of an emergency, the user is easily able to touch it with the left hand, which is the hand the user generally uses. In this case, the vertical length of the emergency stop button214may be greater than the vertical length of the enable button212. For example, the vertical length of the emergency stop button214may be set to be the same as the vertical length of the touchscreen. Additionally or alternatively, the horizontal length of the emergency stop button214may be greater than the horizontal length of the enable button212. InFIG. 2, the portable terminal110is displayed in a portrait mode, but not limited thereto, and when the portable terminal110is displayed in a landscape mode, the enable button212and the emergency stop button214may be rearranged to be displayed on the left and right sides of the touchscreen, respectively, while the user is holding the portable terminal110.

FIG. 3is a diagram illustrating a central area310in which robot control buttons are displayed, and a border area320in which the enable button and the emergency stop button are displayed according to an exemplary embodiment. The touchscreen of the portable terminal110of the present disclosure may be divided into the central area310and the border area320surrounding the central area. The robot control buttons may be displayed in the central area310of the touchscreen. For example, a jog dial, a mode change switch, a motor key, a lock key, a robot selection key, an operation mode key, a speed key, a cursor key, a stop key, an OK key, and the like may be displayed in the central area310of the touchscreen.

The enable button and the emergency stop button may be displayed in the border area320of the touchscreen. In an embodiment, the user may select the positions at which the enable button and the emergency stop button are displayed in the border area320. In addition, the user may adjust the sizes of the enable button and the emergency stop button, or change the shapes of the enable button and the emergency stop button. That is, the user may freely customize the positions, sizes, and shapes of the enable button and the emergency stop button according to intended use and his or her preferences. The ratio of the central area310and the border area320on the touchscreen, and sizes and shapes thereof may also be changed as selected by the user.

FIGS. 4A and 4Bare diagrams illustrating an example in which the user places an enable button422and an emergency stop button424according to an exemplary embodiment. InFIGS. 4A and 4B, first and second operation steps410,420of the portable terminal110are illustrated. The first operation step410represents, while the border area is divided and displayed into a plurality of guide blocks, receiving a selection of the guide blocks where the enable button and the emergency stop button are to be placed.

In the first operation step410, the user may select one guide block412for the area where the enable button is to be placed. In addition, the user may select three guide blocks414,416, and418for the area where the emergency stop button is to be placed. In an embodiment, the selected guide blocks412,414,416, and418may be highlighted or colored and displayed on the screen such that the guide block selected by the user can be easily noticed. In this case, the guide block in which the enable button is to be placed and the guide block in which the emergency stop button is to be disposed may be displayed in different colors from each other.

The second operation step420represents the enable button422and the emergency stop button424placed according to a user input. As illustrated, the enable button422may be placed on the same area as the guide block412selected by the user. In addition, the emergency stop button424may be placed on the same area as the guide blocks414,416, and418selected by the user.

By selecting the guide block displayed on the screen and placing the enable button and the emergency stop button, the user may determine the placement position, size, shape, and the like of the enable button and the emergency stop button to suit his or her taste.

FIG. 5is a diagram illustrating a placement of an enable button510and an emergency stop button512according to another embodiment of the present disclosure. In an embodiment, the user may change the size and shape of the enable button510and the emergency stop button512of the portable terminal110. For example, the size and shape of each button may be changed by changing the selection of a plurality of guide blocks illustrated inFIGS. 4A and 4B. Additionally or alternatively, the size or shape of the enable button510and the emergency stop button512displayed on the screen may be modified through a touch input, a drag input, and the like.

Depending on users, the part of the portable terminal110that is grabbed by the user may be varied when the user holds the portable terminal110with one or both hands. Accordingly, each user may have different preferred button input position. According to the configuration described above, the user may adjust the positions, sizes, and shapes of the enable button510and the emergency stop button512according to his or her preferences, and accordingly, a user interface optimized for each user can be provided. Accordingly, the user can efficiently control the robot device using the portable terminal110and quickly cope with an emergency situation.

FIG. 6is a diagram illustrating an example in which the portable terminal110detects a fall or shaking of a threshold value or above, and transmits an emergency signal610to the control device120according to an exemplary embodiment. In an embodiment, the portable terminal110may include a motion sensor (e.g., an accelerometer, a gyroscope, a gravity sensor, and the like) for detecting a movement of the portable terminal. For example, the accelerometer measures acceleration generated according to the movement of the portable terminal110, and the measured values are expressed as roll, pitch, and yaw. The roll value indicates the acceleration of the X-axis rotation of the portable terminal110, the pitch value indicates the acceleration of the Y-axis rotation, and the yaw value indicates the acceleration of the Z-axis rotation. In an embodiment, when the accelerometer detects the fall or shaking of the portable terminal110of the threshold value or above, the portable terminal110may generate the emergency signal610and transmit it to the control device120without requiring the user to touch the emergency stop button114. When receiving the emergency signal610from the portable terminal110, the control device120may generate an emergency stop command620for stopping all operations of the robot device130and transmit it to the robot device130. In this case, the robot device130receiving the emergency stop command620may stop all operations.

Additionally or alternatively, when the accelerometer or the gravity sensor detects a preset posture, the portable terminal110may generate an emergency stop command620for stopping all operations of the robot device130and transmit it to the control device120. For example, when the user puts the portable terminal110down on a flat surface or mounts it on a cradle, the portable terminal110may detect such a posture, recognize it as a situation that the user has stopped controlling the robot, and generate the emergency signal610and transmit it to the control device120.

FIG. 7is a diagram illustrating an example in which the portable terminal110detects a gesture710of a user716and transmits an emergency signal720to the control device120according to an exemplary embodiment. According to an embodiment, the portable terminal110may include an image sensor718and a processor for image processing. For example, the image sensor718may include a camera, a camcorder, a web cam, and the like, and may be disposed on a front or back side of the portable terminal110.

As illustrated, the user716may make a predefined gesture710to stop all operations of the robot device130without holding the portable terminal110. In this case, the image sensor718of the portable terminal110may detect the gesture710of the user716. In response to detecting the gesture710of the user716, the portable terminal110may generate the emergency signal720and transmit it to the control device120.

When receiving the emergency signal720, the control device120may generate an emergency stop command730for stopping all operations of the robot device130and transmit it to the robot device130. In this case, the robot device130receiving the emergency stop command730may stop all operations. According to the configuration described above, without requiring the user to touch the emergency stop button714, the portable terminal110may quickly stop the operation of the robot device130based on the gesture710of the user716in an emergency situation.

FIG. 8is a diagram illustrating an example in which the portable terminal110detects a voice command810of a user816and transmits an emergency signal820to the control device120according to an exemplary embodiment. In an embodiment, the portable terminal110may be configured to include a sound sensor that receives a voice command of the user and a processor that processes the voice command. The portable terminal110may receive a voice command810from the user816and transmit a command corresponding to the voice command810to the control device120.

As illustrated, the user816may utter the predefined voice command810to stop all operations of the robot device130without holding the portable terminal110. For example, when the portable terminal110detects the voice command810such as “Cease”, “No”, “Discontinue”, “Stop”, the portable terminal110may generate the emergency signal820and transmit it to the control device120. Additionally or alternatively, when the portable terminal110detects the onomatopoeic words “Argh”, “Aaargh”, “Aaaiiieeee”, and the like of a certain decibel (dB) or higher, the portable terminal110may generate the emergency signal820and transmit it to the control device120.

When receiving the emergency signal820, the control device120may generate an emergency stop command830for stopping all operations of the robot device130and transmit it to the robot device130. In this case, the robot device130receiving the emergency stop command830may stop all operations. According to the configuration described above, without requiring the user to touch the emergency stop button814, the portable terminal110may quickly stop the operation of the robot device130based on the voice command810of the user816in an emergency situation.

FIG. 9is a flowchart illustrating a method for placing an enable button and an emergency stop button by a user according to an exemplary embodiment. The method for placing buttons may begin at S910, by dividing and displaying the border area of the portable terminal into a plurality of guide blocks. In this case, the portable terminal may receive a first user input for selecting at least one of a plurality of guide blocks on the screen (S920). Here, the first user input may be the one for determining the placement, size and shape of the enable button.

Next, the portable terminal may receive a second user input for selecting at least one of a plurality of guide blocks on the screen (S930). Here, the second user input may be the one for selecting a guide block different from the guide block selected by the first user input. The second user input may be the one for determining the placement, size and shape of the emergency stop button.

Based on the first and second user inputs, the portable terminal may display the enable button and the emergency stop button on the screen. Specifically, the portable terminal may display the enable button on the first area defined by the guide block(s) selected by the first user input (S940). In addition, the portable terminal may display the emergency stop button on the second area defined by the guide block(s) selected by the second user input (S950). According to the configuration described above, it is possible to provide a user interface optimized for each user, since the user may determine the positions, sizes, and shapes of the enable button and the emergency stop button displayed on the screen according to his or her preferences. In addition, it is possible to provide a method for controlling a robot device, which is capable of quickly responding to an emergency situation by, in response to detecting a registered posture of the user terminal, impact and shaking of the user terminal, a user gesture, a voice command, and the like by an accelerometer, an image sensor, a sound sensor, and the like of the user terminal, transmitting an emergency signal for stopping the operation of the robot device.

The preferred embodiments of the present invention described above are disclosed for purposes of illustration, and those skilled in the art with ordinary knowledge of the present invention will be able to make various modifications, changes and additions within the spirit and scope of the present invention, and such modifications, changes and additions should be construed to be included in a scope of the claims.

When those skilled in the art to which the present invention belongs, various substitutions, modifications, and changes are possible within the scope of the technical spirit of the present invention, and thus the present invention is not limited by the embodiments described above and the accompanying drawings.