INFORMATION PROCESSING APPARATUS AND ROBOT ARM CONTROL SYSTEM

An information processing apparatus supplies information to a robot arm that includes a holding mechanism for holding an article by suctioning the article and a moving mechanism for moving the holding mechanism. The apparatus comprises a communication interface and a controller. The controller calculates a suction point at which the holding mechanism attempts to suction the article. The suction point is calculated using an image captured by a camera. When the controller cannot calculate the suction point using the image, the controller acquires a selected region from an operation terminal operated by an operator, then recalculates the suction point on the basis of the selected region, and attempts suctioning at suction point calculated using the selected region.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority from Japanese Patent Application No. 2017-209394, filed Oct. 30, 2017, the entire contents of which are incorporated herein by reference.

FIELD

Embodiments described herein relate generally to an information processing apparatus and a robot arm control system.

BACKGROUND

A sorting system for delivering articles, such as packages or parcels, requires sorting packages according to intended delivery destinations. In one sorting system a holding mechanism holds an article by suction, and a robot arm moves the holding mechanism.

In a sorting system including a robot arm, a shape of an article is recognized on the basis of an image, and a suction point at which the holding mechanism can attach to the article is determined on the basis of a recognition result. However, when there is an error in recognition, there is a possibility that a suction point might not be determinable, or in some cases the holding mechanism might be moved to an intended suction point, but the holding mechanism may be unable to attach to the article as intended. In such cases, it can be necessary to stop operations of the robot arm and have an operator adjust the position of the article.

DETAILED DESCRIPTION

One embodiment of an information processing apparatus for a robot arm includes a communication interface connected to the robot arm. The robot arm includes a holding mechanism for holding an article by suction of the article and arm mechanism for moving the holding mechanism. A controller is configured to calculate a suction point on the article at which the holding mechanism can suction the article. The suction point is calculated on the basis of an image of the article captured by a camera from a predetermined range. When the suction point is successfully calculated, calculated suction point is indicated to the robot arm via the communication interface. When the suction point cannot be calculated by the controller, the controller is further configured to supply the image to an operation terminal as image information and to acquire a selected region from the operation terminal that generates the selected region in the image information using operator input and then recalculate the suction point using the selected region. The recalculated suction point is supplied to the robot arm via the communication interface.

Hereinafter, example embodiments will be described with reference to the drawings.

FIG. 1is a diagram illustrating a schematic configuration example of a sorting system1.

The sorting system1is a system sorting articles3, which are sorting targets mounted in a source location such as cart2, into predetermined sorting destinations. The sorting destinations are, for example, belt conveyers4. The sorting destinations are not limited to the belt conveyers4, and may be sorting baskets or workbenches.

The sorting system1includes a robot arm11, an information processing apparatus12, and an operation terminal13. The sorting system1further includes a camera14. The robot arm11, the information processing apparatus12, the operation terminal13, and the camera14can perform communication with each other via a network15.

First, a description will be made of a configuration of the robot arm11.

The robot arm11is a device which picks up the article3from the cart2and supplies the article to a sorting destination. The robot arm11includes a holding mechanism21, an arm mechanism22, a contact sensor23, a pressure sensor24, and a controller25.

The holding mechanism21includes, for example, at least one suction pad for suctioning the article3. The suction pad includes a pad portion which is brought into contact with a surface of the article3and communicates with a vacuum pump (not specifically illustrated) controlled by the controller25.

The arm mechanism22is a mechanism moving the holding mechanism21. The arm mechanism22includes a plurality of arms, and joint mechanisms connecting the plurality of arms to each other. The joint mechanisms are moved by actuators controlled by the controller25, so as to drive the arms.

The contact sensor23is a sensor sensing stress applied to the holding mechanism21. The contact sensor23senses, for example, stress applied to the holding mechanism21from a vertical direction. The contact sensor23transmits a sensing result to the controller25. Specifically, in a case where the contact sensor23senses stress applied to the holding mechanism21from the vertical direction, an ON signal is supplied to the controller25.

The pressure sensor24is a sensor sensing pressure in the suction pad of the holding mechanism21. The pressure sensor24senses, for example, pressure of a connection portion where the suction pad of the holding mechanism21is connected to the vacuum pump. The pressure sensor transmits a sensing result to the controller. Specifically, in a case where the pressure sensor24senses that pressure in the suction pad of the holding mechanism21is negative pressure, an ON signal is supplied to the controller25.

The controller25controls operations of the holding mechanism21and the arm mechanism22. The controller25includes a memory (not illustrated), and a calculation element (not illustrated) controlling operations of the holding mechanism21and the arm mechanism22by executing a program stored in the memory. The controller25may include an operation sequencer or the like.

The camera14includes a lens, and an imaging element which converts light imaged by the lens into image data. An angle of view of the lens of the camera14is adjusted such that the inside of the cart2with the articles3to be picked up by the robot arm11can be imaged. For example, the lens of the camera14is disposed such that a range in which the article3can be suctioned by the holding mechanism21is included in an angle of view. The camera14supplies an image generated by imaging the inside of the cart2to the information processing apparatus12via the network15. As the camera, for example, a 3D (or stereoscopic) camera which can measure information regarding a distance to the article3may be used.

FIG. 2is a diagram illustrating a configuration example of the information processing apparatus12.

The information processing apparatus12is an apparatus supplying coordinate information to the robot arm11. The coordinate information is a coordinate in a surface (horizontal surface) orthogonal to the vertical direction, and is information indicating a suction point which is a position where the suction pad of the holding mechanism21of the robot arm11suctions the article3. In other words, the information processing apparatus12gives an instruction for a suction point to the robot arm11.

For example, the information processing apparatus12is configured with an apparatus such as a personal computer (PC) capable of processing data and storing data.

The information processing apparatus12includes a communication interface31and a control unit32.

The communication interface31is used to communicate with other apparatuses other than the information processing apparatus12. The communication interface31includes terminals and circuits corresponding to a communication standard for communicating with the robot arm11, the operation terminal13, and the camera14via the network15. The communication interface31communicates with the robot arm11, the operation terminal13, and the camera14under the control of the control unit32.

The control unit32is a processing unit performing various processes. The control unit32includes a processor33and a memory34.

The processor33is a calculation element performing a calculation process. The processor33is configured with, for example, a CPU. The processor33performs various processes on the basis of a program stored in the memory34.

The memory34is a storage device storing a program and data. The memory34includes at least one of, for example, a ROM which is a read only nonvolatile memory, a RAM which temporarily stores data, and a storage which stores data.

Next, a description will be made of a configuration of the operation terminal13.

FIG. 3is a diagram illustrating a configuration example of the operation terminal13.

The operation terminal13is a device supplying information based on an operation to the information processing apparatus12. The operation terminal13includes a communication interface41, a control unit42, and a touch panel43.

The communication interface41is used to communicate with apparatuses other than the operation terminal13. The communication interface41includes terminals and circuits corresponding to a communication standard for communicating with the robot arm11, the information processing apparatus12, and the camera14via the network15. The communication interface41communicates with the robot arm11, the information processing apparatus12, and the camera14under the control of the control unit42.

The control unit42is a processing unit performing various processes. The control unit42includes a processor44and a memory45.

The processor44is a calculation element performing a calculation process. The processor44is configured with, for example, a CPU. The processor44performs various processes on the basis of a program stored in the memory45.

The memory45is a storage device storing a program and data. The memory45includes at least one of, for example, a ROM which is a read only nonvolatile memory, a RAM which temporarily stores data, and a storage which stores data.

The touch panel43is a device which displays a screen and generates an operation signal based on an operation. The touch panel43includes a display46and a touch sensor47integrally configured with each other. The operation terminal13may include a display on which a screen is displayed and an operation unit which generates an operation signal based on an operation, instead of the touch panel43. The operation unit may be any input unit such as a mouse, a trackball, a keyboard, and a trackpad.

The display46displays a screen on the basis of display data (screen data) supplied from the control unit42or a graphic controller.

The touch sensor47generates an operation signal indicating a position touched by an operator operating the operation terminal13on a screen displayed on the display46.

FIG. 4is a flowchart illustrating an operation of the robot arm11.

The controller25of the robot arm11moves the arm to be outside of the angle of view (that is, beyond of the imaging range) of the lens of the camera14(step S11). For example, the controller25causes the actuator to move the joint mechanism of the arm mechanism22, so as to drive the arm mechanism22, and thus moves the arm mechanism22and the holding mechanism21out of the imaging range. The camera14acquires an image at a timing at which the arm mechanism22and the holding mechanism21are moved out of the imaging range, and transmits the image to the information processing apparatus12.

The controller25moves the arm mechanism22and the holding mechanism21out of the imaging range, and then waits for coordinate information to be supplied from the information processing apparatus12(step S12).

In a case where coordinate information is supplied from the information processing apparatus12(YES in step S12), the controller25moves the suction pad of the holding mechanism21on the basis of the coordinate information (step S13). For example, the controller25controls the arm mechanism22to move the suction pad of the holding mechanism21to a suction point indicated by the coordinate information. More specifically, the controller25controls the arm mechanism22such that the suction point indicated by the coordinate information will overlap a reference point in the suction pad of the holding mechanism21. The reference point in the suction pad may be, for example, the center (centroid) of the suction pad, and may be any point in the suction pad.

The controller25controls the arm mechanism22to move down the holding mechanism21at the suction point (step S14).

The controller25determines whether or not a contact sensing result of the contact sensor23is in an ON state (step S15). The controller25determines whether or not the suction pad of the holding mechanism21has been brought into contact with a surface of the article3on the basis of the sensing result of the contact sensor23. For example, in a case where an ON signal is received from the contact sensor23, the controller25determines that the suction pad of the holding mechanism21has been brought into contact with the surface of the article3. In other words, the controller25determines whether or not the suction pad is contacting the article3.

In a case where an ON signal is not received from the contact sensor23(NO in step S15), the controller25determines whether or not the maximum stroke has occurred (step S16). In a case where the holding mechanism21cannot be moved down any farther by the arm mechanism22, the controller25determines that the maximum stroke has occurred.

In a case where it is determined that the maximum stroke has not occurred (NO in step S16), the controller25proceeds to a process in step S14, and controls the arm mechanism22to move down the holding mechanism21again. Consequently, the controller25moves the holding mechanism21downward until the suction pad is brought into contact with the surface of the article3.

In a case where it is determined that the maximum stroke occurs (YES in step S16), the controller25transmits an contact sensing error indicating that the suction pad is not brought into contact with the article3to the information processing apparatus12(step S17), and proceeds to a process in step S11. In other words, in a case where the suction pad of the holding mechanism21is not brought into contact with the surface of the article3even if the holding mechanism21is moved down to the maximum stroke at the suction point for which the instruction is given from the information processing apparatus12, the controller25transmits the contact sensing error to the information processing apparatus12. The controller25moves the arm mechanism22and the holding mechanism21beyond the angle of view of the camera14, and waits for the next coordinate information to be received.

In a case where an ON signal is received from the contact sensor23in step S15(YES in step S15), the controller25performs a suction process by using the holding mechanism21(step S18). The controller25controls the suction pad suctioning of the surface of the article3by operating the vacuum pump connected to the suction pad of the holding mechanism21. For example, the controller25causes the inside of the suction pad to become vacuum using the vacuum pump while the suction pad is brought into contact with the article, and thus causes the suction pad to apply suction the surface of the article3.

The controller25determines whether or not the pressure in the suction pad of the holding mechanism21is negative pressure on the basis of a sensing result (pressure sensing result) in the pressure sensor24(step S19). In other words, the controller25determines whether or not the suction pad of the holding mechanism21applies suctions the surface of the article3on the basis of the sensing result in the pressure sensor24. For example, in a case where an ON signal is received from the pressure sensor24, the controller25determines that the inside of the suction pad of the holding mechanism21is at vacuum, and thus the suction pad suctions the surface of the article3.

In a case where it is determined that the pressure in the suction pad of the holding mechanism21is not negative pressure

(NO in step S19), the controller25transmits a pressure sensing error to the information processing apparatus12indicating that the suction pad is not applying suction the article3(step S20), and proceeds to step S11. In other words, in a case where the suction pad does not suction the surface of the article3even though a suction process is performed with the suction pad of the holding mechanism21in contact with the surface of the article3, the controller25transmits the pressure sensing error to the information processing apparatus12. The controller25moves the arm mechanism22and the holding mechanism21beyond the angle of view of the camera14, and waits for the next coordinate information to be received. In a case where it is continuously determined for a predetermined time whether or not the suction pad of the holding mechanism21is successfully applying suction to the surface of the article3, and an ON signal is not received from the pressure sensor24for the predetermined time, the controller25may transmit a pressure sensing error to the information processing apparatus12.

In a case where the controller25determines that the pressure in the suction pad of the holding mechanism21is negative pressure (YES in step S19), the controller25controls the arm mechanism22to move up the holding mechanism21holding the article3(step S21).

Once the holding mechanism21is moved up to a predetermined height, the controller25controls the arm mechanism22to move the holding mechanism21to a position corresponding to the belt conveyer4associated with a sorting destination (step S22).

Once the holding mechanism21is moved to the position corresponding to the belt conveyer4associated with a sorting destination, the controller25controls the holding mechanism21to release the article3from the suction pad (step S23), transmits a completion notification indicating that sorting of a single article is completed to the information processing apparatus12, and proceeds to the process in step S11. Consequently, the article3released from the suction pad is placed on the belt conveyer4. The controller25moves the arm mechanism22and the holding mechanism21from the view of the camera14, and waits for the next coordinate information to be received.

When a finish instruction is supplied from the information processing apparatus12, the controller25of the robot arm11moves the arm mechanism22and the holding mechanism21to predetermined positions and ends the process inFIG. 4.

FIGS. 5 to 7are flowcharts illustrating an operation of the information processing apparatus12.

The control unit32of the information processing apparatus12waits for an image to be supplied from the camera14(step S31). As described above, the camera14supplies an image of the inside of the cart2mounted with the articles3to be picked up by the robot arm11, to the information processing apparatus12.

FIG. 8is a diagram illustrating an example of an image supplied from the camera14. As illustrated inFIG. 8, a plurality of articles3mounted in the cart2is depicted in the image supplied from the camera14. Since the plurality of articles3are in the cart2in an overlapping manner, an article3having a part that is hidden by another article3is also shown.

When the image is received from the camera14(YES in step S31), the control unit32performs image processing on the received image (step S32). For example, the control unit32generates a mask image.

FIG. 9is a diagram illustrating an example of a mask image. The mask image is an image obtained by extracting (or selecting) a part of the entire image. For example, the control unit32generates a mask image by extracting a region51that matches a surface which is present at the highest position in the cart2using distance information acquired by a distance sensor measuring a distance to the article3.

The control unit32recognizes a shape of the article3which is a sorting target on the basis of mask image (step S33). For example, the control unit32recognizes a contour (region outline) in the mask image as a shape of the article3. The control unit32recognizes a surface surrounded by the contour in the image, that is, the region51depicted inFIG. 9as a suction surface on which a suction process can be performed by the suction pad. The control unit32may recognize a shape of the article3by detecting a specific shape or outline in the image without performing image processing in step S32.

The control unit32determines whether or not there is an article to be sorted on the basis of a shape recognition result on the article3(step S34). For example, in a case where a suction surface can be recognized in step S33and a size of the suction surface is larger than a preset size, the control unit32determines that there is an article which can be sorted. In a case where a suction surface can be recognized in step S33and a size of the suction surface is smaller than the preset size, the control unit32determines that there is no article which can be sorted. In a case where a suction surface cannot be recognized in step S33, the control unit32determines that there is no article which can be sorted.

In a case where it is determined that there is a sortable article (YES in step S34), the control unit32calculates a coordinate of a suction point (step S35). For example, the control unit32calculates the coordinate of the suction point on the basis of the suction surface recognized in step S33. Specifically, the control unit32calculates a coordinate of the center (or the centroid) of the suction surface as the coordinate of the suction point.

The control unit32generates coordinate information indicating the coordinate of the suction point, and transmits the coordinate information to the robot arm11(step S36).

The control unit32determines whether or not a completion notification has been received from the robot arm11(step S37). In a case where it is determined that the completion notification has been received from the robot arm11(YES in step S37), the control unit32proceeds to a process in step S31, and calculates a suction point of the next article3on the basis of an image supplied from the camera14.

In a case where it is determined that the completion notification is not received from the robot arm11(NO in step S37), the control unit32determines whether or not a contact sensing error has been received from the robot arm11(step S38).

In a case where it is determined that the contact sensing error is not received from the robot arm11(NO in step S38), the control unit32determines whether or not a pressure sensing error has been received from the robot arm11(step S39).

In a case where it is determined that the pressure sensing error is not received from the robot arm11(NO in step S39), the control unit32proceeds to the process in step S37, and waits any one of a completion notification, a contact sensing error, and a pressure sensing error to be supplied from the robot arm11.

In a case where it is determined that there is no article which can be sorted in step S34(NO in step S34), or it is determined that a contact sensing error has been received from the robot arm11in step S38(YES in step S38), the control unit32proceeds to a process in step S41illustrated inFIG. 6.

In a case where there is no article3in the cart2, the article3is present in a dead angle of the camera14, or a shape of the article3is not accurately recognized, it is supposed that the control unit32determines that there is no article which can be sorted. For example, as illustrated inFIG. 10, a suction surface52of the article3is wrongly recognized, and thus the article3is not present at a calculated suction point53, the robot arm11cannot cause the suction pad to suction a surface of the article3even if the holding mechanism21is moved down to the maximum stroke. In this case, it is supposed that the control unit32receives a contact sensing error from the robot arm11. In this case, it is necessary that an image is transmitted to the operation terminal13such that an operator of the operation terminal13can check the image, and a suction point can be determined on the basis of the operator's input. Therefore, the control unit32transmits the image received from the camera14and information for requesting selection of a region to the operation terminal13(step S41). For example, the control unit32may supply an image obtained by performing image processing on the image received from the camera14to the operation terminal13. The control unit32may supply another image which is acquired by the camera14to the operation terminal13.

In the operation terminal13, region information which is information indicating a specific region on an image is provided through a region input process which will be described later. The operation terminal13transmits the generated region information to the information processing apparatus12.

The control unit32determines whether or not the region information has been received from the operation terminal13(step S42). The control unit32waits for the region information to be received from the operation terminal13.

In a case where it is determined that the region information has been received from the operation terminal13(YES in step S42), the control unit32calculates a coordinate of a suction point on the basis of the region information (step S43). For example, the control unit32calculates the coordinate of the suction point on the basis of the specific region on the image indicated by the region information. Specifically, the control unit32calculates a coordinate of the center (or the centroid) of the specific region as the coordinate of the suction point.

The control unit32generates coordinate information indicating the coordinate of the suction point calculated in step S43, transmits the coordinate information to the robot arm11(step S44), and proceeds to the process in step S37inFIG. 5. Consequently, in a case where it is determined that there is no article which can be sorted, or the suction pad and the article3are not brought into contact with each other at a position corresponding to a suction point, a coordinate of a suction point can be calculated again on the basis of a region which is determined according to an operation of the operator. Therefore, even if poor recognition occurs, the information processing apparatus12can determine an accurate suction point without stopping an operation of the robot arm11.

In a case where it is determined that a pressure sensing error is received from the robot arm11in step S39(YES in step S39), the control unit32proceeds to a process in step S51illustrated inFIG. 7.

In a case where there is a gap between the suction pad and the article3, the pressure sensor24does not sense negative pressure even when a suction process is performed, and a notification of a pressure sensing error will be received from the robot arm11. For example, in a case where the surface of the article3is uneven, there may be a gap between the suction pad and the uneven surface of article3. In this case, it is necessary that an image be transmitted to the operation terminal13such that the operator of the operation terminal13can check the image, and the operator can then select a position where the article3might be more easily suctioned. Therefore, the control unit32transmits the image received from the camera14, and information for requesting that a suction point be selected to the operation terminal13(step S51). For example, the control unit32may supply an image obtained by performing image processing on the image received from the camera14to the operation terminal13. The control unit32may also supply another, new image acquired by the camera14to the operation terminal13.

In the operation terminal13, suction point information indicating a certain coordinate (suction point) on the image is generated by a suction point input process which will be further described. The operation terminal13transmits the generated suction point information to the information processing apparatus12.

The control unit32determines whether or not the suction point information has been received from the operation terminal13(step S52). The control unit32waits for the suction point information to be received from the operation terminal13.

In a case where it is determined that the suction point information has been received from the operation terminal13(YES in step S52), the control unit32determines a coordinate of a suction point on the basis of the suction point information (step S53). For example, the control unit32determines the coordinate indicated by the suction point information as the coordinate of the suction point.

The control unit32generates coordinate information indicating the coordinate of the suction point calculated in step S53, transmits the coordinate information to the robot arm11(step S54), and proceeds to the process in step S37inFIG. 5. Consequently, in a case where the suction pad does not suction the article3in the suction process, a coordinate of a suction point can be calculated again according to an operation of the operator. Therefore, even in a case where there is a region not appropriate for suction in the surface of the article3, the information processing apparatus12can determine an accurate suction point without stopping an operation of the robot arm11.

In a case where a finish instruction is supplied from the operation terminal13, the control unit32of the information processing apparatus12supplies the finish instruction to the robot arm11so as to finish the processes inFIGS. 5 to 7.

In a case where it is determined that a completion notification has not been received from the robot arm11in step S37inFIG. 5then regardless of retransmission of coordinate information to the robot arm11through the processes in steps S41to S44inFIG. 6or steps S51to S54inFIG. 7, the control unit32may send a notification of an error without proceeding to the processes in steps S38and S37. For example, the control unit32transmits and displays a screen indicating a failure in holding to the operation terminal13. For example, the control unit32may cause a display device or a speaker connected to the network15of the sorting system1to notify the operator of a failure in holding.

FIGS. 11 and 12are flowcharts illustrating an operation of the operation terminal13.

The control unit42of the operation terminal13waits for an image to be supplied from the information processing apparatus12(step S61).

If an image is supplied from the information processing apparatus12(YES in step S61), the control unit42determines whether there has been a request for selection of a region or selection of a suction point (step S62). For example, in a case where information for requesting selection of a region is transmitted from the information processing apparatus12along with an image, the control unit42determines that there has been a request for selection of a region. For example, in a case where information for requesting selection of a suction point is transmitted from the information processing apparatus12along with an image, the control unit42determines that there has been a request for selection of a suction point.

When a request for selection of a region is made, the control unit42performs a region input process . In this case, the control unit42first displays the supplied image on the display46(step S63).

The control unit42also displays information for prompting input of a region on the display46(step S64). The operator inputs a region by performing an operation of tracing some region on the touch panel43. The control unit42may further display a finish button for stopping (finishing) an operation of the robot arm11on the display46without the inputting of a region by the operator.

For example, the information for prompting input of a region is information for prompting a contour of the article3to be drawn. In this case, the operator inputs a region by performing an operation of tracing a contour of the article3displayed on the display46of the touch panel43.

FIG. 13is a diagram for explaining the result of the operator performing of the tracing of a contour of the article3displayed on the display46. The control unit42specifies a region54on the image on the basis of a trajectory of touch positions on the touch sensor47when the operator performs the tracing operation. The information processing apparatus12may calculate a suction point55on the basis of the region54.

For example, the information for prompting drawing of a region is information for prompting an identification of a region which can be suctioned on the article3. In this case, the operator inputs a region by performing an operation of tracing a contour of any region which is determined as being able to be suctioned in the article3displayed on the display46of the touch panel43.

FIG. 14is a diagram for explaining a result of the operator performing a tracing of a contour of any region selected as being able to be suctioned on the article3displayed on the display46. The control unit42specifies a region56on the image on the basis of a trajectory of touch positions on the touch sensor47when the operator performs the operation of tracing the contour of the region. The information processing apparatus12may calculate a suction point57on the basis of the region56.

The control unit42determines whether or not an operation for inputting a region has been performed (step S65). For example, in a case where an input operation of surrounding a region has been performed on the touch sensor47while the display in steps S63and S64is performed, the control unit42determines whether or not an operation of inputting a region has been performed, and the region as illustrated inFIG. 13 or 14is selected.

In a case where it is determined that an inputting of a region has been performed and the region is specified (YES in step S65), the control unit42generates region information indicating a coordinate of the specified region on the image (step S66). For example, the region information may be vector data generated on the basis of a trajectory of touch positions on the touch sensor47, and may be raster data corresponding to a position of the region.

The control unit42transmits the generated region information to the information processing apparatus12(step S67), and finishes the process.

In a case where it is determined that an operation of inputting a region has not been performed in step S65(NO in step S65), the control unit42determines whether or not an operation of the robot arm11is finished (step S68). For example, in a case where a predetermined time elapses in a state in which a region input operation is not performed, the control unit42determines that an operation of the robot arm11is finished. For example, in a case where a finish button displayed on the touch panel43is selected, the control unit42may determine that an operation of the robot arm11is finished.

In a case where it is determined that an operation of the robot arm11is not finished (NO in step S68), the control unit42proceeds to the process in step S64, and determines again whether or not a region has been input.

In a case where it is determined that an operation of the robot arm11is finished (YES in step S68), the control unit42supplies a finish instruction to the information processing apparatus12(step S69), and finishes the process inFIG. 11.

In a case where it is determined that there is a request for selection of a suction point in step S62, the control unit42performs a suction point input process. In this case, first, the control unit42displays the supplied image on the display46(step S71).

The control unit42displays information for prompting input of a suction point on the display46(step S72). The operator inputs a suction point by performing an operation of touching any suction point on the touch panel43. The control unit42may further display a display button (finish button) for stopping (finishing) an operation of the robot arm11on the display46without inputting a suction point.

For example, the information for prompting input of a suction point is information for prompting selection of a position at which suction is possible by the suction pad in a surface of the article3to be sorted. In this case, the operator inputs a suction point by recognizing a state of the surface of the article3displayed on the display46of the touch panel43from the image and touching any position on the screen.

FIG. 15is a diagram for explaining a result of the operator performing an operation of selecting a suction point in the surface of the article3displayed on the display46of the touch panel43. The control unit42specifies a suction point58on the image on the basis of a touch position on the touch sensor47.

The control unit42determines whether or not an operation of inputting a suction point is performed (step S73). For example, in a case where a touch operation is performed on the touch sensor47in a state in which the display in steps S71and S72is performed, the control unit42determines that the suction point as illustrated inFIG. 15is specified.

In a case where it is determined that the suction point is specified (YES instep S73), the control unit42generates suction point information indicating a coordinate of the specified suction point on the image (step S74). The suction point information is data regarding a coordinate corresponding to the touch position on the touch sensor47.

The control unit42transmits the generated suction point information to the information processing apparatus12(step S75), and finishes the process.

In a case where the suction point information (coordinate information) supplied from the information processing apparatus12is received, the controller25of the robot arm11controls the arm mechanism22to move the holding mechanism21at a position where the reference point in the suction pad of the holding mechanism21overlaps the suction point indicated by the suction point information. The controller25moves the holding mechanism21downward, and determines whether or not the suction pad of the holding mechanism21has been brought into contact with a surface of the article3. In a case where the suction pad of the holding mechanism21has been brought into contact with the surface of the article3, the controller25performs a suction process by using the holding mechanism21such that the suction pad suctions the surface of the article3.

When a suction point is not specified in step S73(NO in step S73), the control unit42determines whether or not an operation of the robot arm11is finished (step S76). For example, in a case where a predetermined time elapses before a suction point input operation is performed, the control unit42determines that an operation of the robot arm11is finished (timed out). Also, when a finish button displayed on the touch panel43is selected, the control unit42may determine that an operation of the robot arm11is finished.

When an operation of the robot arm11is not finished (NO in step S76), the control unit42proceeds to the process in step S73, and determines again whether or not a suction point has been input.

In a case where it is determined that an operation of the robot arm11is finished (YES in step S76), the control unit42supplies a finish instruction to the information processing apparatus12(step S77), and ends the process inFIG. 12.

As described above, when it is determined that there is no sortable article according to image recognition results or that the suction pad cannot be brought into contact with the article3at a position corresponding to a suction point, the information processing apparatus12supplies information for requesting input of a region and an image to the operation terminal13. The operation terminal13displays the image on the touch panel43, and prompts the operator to select a region. Consequently, the operator inputs a region selection corresponding to a surface of the article3on the touch panel43. The information processing apparatus12calculates a coordinate of a suction point again on the basis of the region which has been input by the operator, and supplies the coordinate of the suction point to the robot arm11. As a result, even in a case where poor image recognition occurs, the information processing apparatus12can cause the robot arm11to perform operations on the basis of the recalculated suction point.

In a case where the operator traces a contour of the article3, the operation terminal13can specify a region corresponding to a suction surface of the article3on an image. Consequently, even in a case where a suction surface is not accurately recognized in image processing by the information processing apparatus12, the operation terminal13can be used to teach an accurate suction surface to the information processing apparatus12.

For example, in a case where there is a region that is not appropriate for suctioning on the article3, the operator can perform an operation of tracing a region excluding the inappropriate region. In this case, the operation terminal13can be used to specify a region by avoiding the inappropriate region. Consequently, the operation terminal13can be used to teach which regions can be more easily suctioned by the robot arm11to the information processing apparatus12.

As described above, in a case where a suction process fails, the information processing apparatus12supplies information for requesting input of a suction point and an image to the operation terminal13. The operation terminal13displays the image on the touch panel43, and prompts the operator to input a suction point. Consequently, the operator inputs a position where the suction pad performs suction in a surface of the article3on the touch panel43. The information processing apparatus12supplies a coordinate corresponding to the suction point which is input by the operator to the robot arm11as coordinate information. As a result, even in a case where a suction process fails, the information processing apparatus12can continuously cause the robot arm11to perform an operation by using the suction point which is input by the operator.

In the embodiment, the holding mechanism21of the robot arm11holds the article3by the suction pad suctioning a surface of the article3, but this configuration is only an example. The holding mechanism21of the robot arm11may hold the article3by interposing the article3among two or more contact points. In this case, the suction point is replaced with two or more contact points. The contact points are not required to be points on a contour of the article3unlike the suction point. In this case, the operator inputs a contour of the article3by tracing the contour of the article3on an image on the operation terminal13. The operation terminal13teaches information indicating the contour of the article3to the information processing apparatus12, and thus the information processing apparatus12recalculates contact points. The operator may input two or more contact points on a contour of the article3on an image on the operation terminal13. The operation terminal13teaches information indicating contact points where the article3can be held to the information processing apparatus12, so as to cause the robot arm11to continuously perform sorting.