Source: http://patents.com/us-9889563.html
Timestamp: 2018-09-23 00:27:37
Document Index: 478461577

Matched Legal Cases: ['art 615', 'art 615', 'art 615', 'art 615', 'art 615', 'art 615', 'art 615', 'art 615']

US Patent # 9,889,563. Systems and methods to facilitate human/robot interaction - Patents.com
United States Patent 9,889,563
Stubbs , et al. February 13, 2018
Stubbs; Andrew (Waltham, MA), Verminski; Matthew David (North Andover, MA), Caldara; Stephen (Cambridge, MA), Shydo, Jr.; Robert Michael (Pelham, NH)
Family ID: 1000002596320
15/483,240
14660161 Mar 17, 2015 9649766
Current CPC Class: B25J 9/1666 (20130101); B25J 9/1676 (20130101); G05D 1/0234 (20130101); G05D 1/0214 (20130101); G06K 7/10376 (20130101); G05B 2219/40202 (20130101); G05D 2201/0216 (20130101); Y10S 901/01 (20130101)
5198800 March 1993 Tozawa et al.
5570992 November 1996 Lemelson
6061617 May 2000 Berger et al.
6687571 February 2004 Byrne et al.
6784800 August 2004 Orzechowski
6845311 January 2005 Stratton et al.
7079931 July 2006 Sahm et al.
7310439 December 2007 Silverstein
7353089 April 2008 McEvoy
7400959 July 2008 Price et al.
7863848 January 2011 Sjoberg
7890235 February 2011 Self
8115650 February 2012 Dasilva
8179257 May 2012 Allen, Jr.
8195344 June 2012 Song
8514081 August 2013 Cristache
8970363 March 2015 Kraimer
8972045 March 2015 Mountz et al.
9041561 May 2015 Wallace et al.
9122276 September 2015 Kraimer
9126334 September 2015 Nakamura
2004/0221790 November 2004 Sinclair et al.
2004/0249510 December 2004 Hanson
2005/0107934 May 2005 Gudat
2008/0009967 January 2008 Bruemmer
2009/0033269 February 2009 Griessnig
2009/0141938 June 2009 Lim et al.
2009/0157228 June 2009 Hong et al.
2011/0010024 January 2011 Salisbury
2011/0227748 September 2011 Schaible et al.
2012/0182392 July 2012 Kearns et al.
2012/0326837 December 2012 Ajay
2013/0108992 May 2013 Buelthoff et al.
2014/0160250 June 2014 Pomerantz et al.
2014/0277691 September 2014 Jacobus
2014/0326580 November 2014 Carpenter et al.
2015/0158182 June 2015 Farlow et al.
2015/0332463 November 2015 Galera et al.
2016/0057925 March 2016 Letsky
2016/0165795 June 2016 Balutis et al.
2016/0271800 September 2016 Stubbs et al.
2016/0274586 September 2016 Stubbs et al.
102013001987 Aug 2014 DE
Ettouhami, et al, "Human-Robot Collision Avoidance with RFID Sensors Using Fuzzy Logic and Extended Kalman Filter", Applied Mathematical Sciences, Jan. 1, 2013, pp. 2555-2567, retrieved from the internet at URL://http://www.m-hikari.com/ams/ams-2013/ams-49-52-2013/ettouhamiAMS49-- 52-2013.pdf, retrived on Apr. 23, 2016. cited by applicant .
Office action for U.S. Appl. No. 14/660,259, dated Mar. 11, 2016, Stubbs et al., "System and Methods to Facilitate Human/Robot Interaction ", 15 pages. cited by applicant .
Office action for U.S. Appl. No. 14/660,161, dated Aug. 11, 2016, Stubbs et al., "Systems and Methods to Facilitate Human/Robot Interaction", 15 pages. cited by applicant .
PCT Search Report and Written Opinion dated May 19, 2016 for PCT application No. PCT/US2016/022443, 14 pages. cited by applicant .
Shen, et al., "Safe Assembly Motion--A Novel Approach for Applying Human-Robot Co-operation in Hybrid Assembly Systems", 2013 IEEE International Conference on Mechantronics and Automation, IEEE, Aug. 4, 2013, pp. 7-12, retrieved on Oct. 2, 2013. cited by applicant .
Vicentini, et al., "Wireless Sensor Networks and Safe Protocols for user tracking in Human-Robot Cooperative workspaces", 2014 IEEE 23rd International Symposium on Industrial Electronics (ISIE), IEEE, Jun. 1, 2014, pp. 1274-1279, retrieved on Jul. 24, 2014. cited by applicant.
This disclosure is a continuation of, and claims priority under 35 U.S.C. .sctn. 120 to, U.S. patent application Ser. No. 14/660,161, filed Mar. 17, 2015, of the same title, which is incorporated herein by reference as if fully set forth below.
1. A system comprising: a robot to perform one or more tasks in a workspace, the robot comprising: one or more drive mechanisms to move the robot throughout the workspace; and a processor in communication with at least the one or more drive mechanisms; a wearable device comprising one or more short range transmission tags, the wearable device associated with one of a worker or the robot; and a short range transmission reader configured to read the one or more short range transmission tags and associated with the other of the worker or the robot, the short range transmission reader to detect a short range transmission interaction; wherein the processor instructs the robot to take evasive action based at least in part on a detection of the short range transmission interaction.
3. The system of claim 2, wherein the short range transmission reader refrains from sending the signal to the processor based at least in part on the short range transmission reader no longer reading at least one of the one or more short range transmission tags; and wherein the processor instructs the robot to resume normal operation based at least in part on the processor no longer receiving the signal from the short range transmission reader.
5. The system of claim 1, wherein the short range transmission reader comprises a radio frequency identification (RFID) reader; and wherein the one or more short range transmission tags comprise one or more RFID tags.
6. The system of claim 1, further comprising: a transceiver associated with the short range transmission reader for wirelessly communicating with a central control; wherein the short range transmission reader sends a first signal to the central control via the transceiver to indicate a short range transmission interaction; and wherein the central control sends a second signal to the processor to indicate the short range transmission interaction.
7. A method comprising: receiving, at a robot, a first route from a central control, the first route including at least a first path to a destination to perform a task in a workspace; sending a signal, from a processor to one or more drive mechanisms on the robot, to follow the first route; receiving, at the robot, a signal from the central control that a worker has entered the workspace; receiving, at the robot, a second route from the central control in response to a signal indicating a short range transmission interaction between a short range transmission reader and one or more short range transmission tags; and sending a signal, from a processor to one or more drive mechanisms on the robot, to follow the second route; wherein the short range transmission reader is associated with the one of the worker or the robot; and wherein the one or more short range transmission tags are associated with the other of the worker or the robot.
8. The method of claim 7, further comprising: receiving, at the robot, a third route from the central control in response to a signal indicating a continued short range transmission interaction between a short range transmission reader and one or more short range transmission tags; and sending a signal, from a processor to one or more drive mechanisms on the robot, to follow the third route.
9. The method of claim 7, wherein the first route from the central control further comprises instructions to cause the robot to travel at a first speed; wherein the second route from the central control further comprises instructions to cause the robot to travel at a second speed; and wherein the second speed is slower than the first speed.
10. The method of claim 7, further comprising: receiving, at the robot, instructions to stop from the central control in response to a signal indicating a continued short range transmission interaction.
11. The method of claim 7, wherein the one or more short range transmission tags are associated with the worker; and wherein the short range transmission tags comprise passive radio frequency identification (RFID) tags.
12. A method comprising: receiving, at a robot, routing instructions from a central control, including at least a destination, to perform a task in a workspace; receiving, at the robot, a signal from the central control that a worker has entered the workspace; sending a signal, from a processor to one or more drive mechanisms on the robot, to take evasive action in response to a signal indicating a short range transmission interaction between a short range transmission reader and one or more short range transmission tags; wherein the short range transmission reader is associated with one of the worker or the robot; and wherein the one or more short range transmission tags are associated with the other of the worker or the robot.
13. The method of claim 12, wherein the one or more short range transmission tags are mounted on the robot and the short range transmission reader is associated with the worker, and further comprising: detecting, with a processor on the robot, that the one or more short range transmission tags mounted on the robot are being read by the short range transmission reader associated with the worker indicating the short range transmission interaction.
15. The method of claim 12, wherein the short range transmission reader is mounted on the robot and the one or more short range transmission tags are associated with the worker, and further comprising: detecting, with the short range transmission reader on the robot, the one or more short range transmission tags associated with the worker indicating a short range transmission interaction.
16. The method of claim 12, wherein sending the signal, from the processor to the one or more drive mechanisms on the robot, to take evasive action further comprises: sending a first signal, from the processor to the one or more drive mechanisms on the robot, to take a first evasive action in response to a first signal indicating a short range transmission interaction between the short range transmission reader and a first group of one or more short range transmission tags; and sending a second signal, from the processor to the one or more drive mechanisms on the robot, to take a second evasive action in response to a second signal indicating a short range transmission interaction between the short range transmission reader and a second group of one or more short range transmission tags; wherein the second group is larger than the first group.
17. The method of claim 16, wherein the first evasive action comprises one of slowing down or rerouting; and wherein the second evasive action comprises stopping.
18. The method of claim 12, further comprising: receiving a first instruction from the central control comprising information related to a virtual work zone around the worker in response to the short range transmission interaction.
20. The method of claim 19, wherein the evasive action further comprises at least one of: a first evasive action when the robot enters the outer work zone; a second evasive action when the robot enters the intermediate work zone; or a third evasive action when the robot enters the inner work zone.
While other short range transmissions technologies besides RFID, such as Bluetooth.RTM. and near field communications (NFC), could be used, one advantage of RFID tags is that they are inexpensive. In this manner, redundancy can be provided simply by using multiple RFID tags. When incorporated into a disposable garment, such as a vest or a baseball cap, for example, the whole garment can be replaced if the RFID tags have failed, or are failing. In addition, each garment and/or each robot can include multiple RFID tags to ensure readability from multiple angles and orientations. In this manner, regardless of the relative motion and orientation between the robot and the worker, at least one tag can be read and identified.
In some examples, the robot 120 can send a detection signal to the central control that an RFID interaction exists. The central control can then send instructions to the robot establishing one or more work zones around the worker 102 and/or the robot 120. In other examples, a processor on the robot 120 can detect and process the RFID interaction. Regardless, in response, the robot 120 can take an appropriate "evasive" action.
In some examples, the robot 120 can simply stop until all RFID tags (or RFID readers) associated with a worker 102 are no longer in range of the RFID reader (or RFID tags) on the robot 120. In other examples, the robot 120 can detect its distance from the worker--using RFID tags, fiducials, or other means--as discussed below--to establish multiple, concentric work zones 105. In this manner, the robot 120 can try to reroute around the worker 102 upon initial contact with an outer work zone 105a (e.g., 10 feet), for example, slow down upon contact with an intermediate work zone 105b, and then stop upon detection of an inner work zone 105c (e.g., 5 feet). At 145, when the robot 120 (1) ceases to detect, with the RFID reader, RFID tags associated with the worker 102 (or any worker 102), or (2) ceases to detect a reader associated with the worker writing to its RFID tags, it can return to normal operation (e.g., continue on its route at a normal speed).
Regardless of the reason, it is desirable to prevent incidents between workers 102 and robots 120. To this end, examples of the present disclosure can comprise an inventory control system 200 for establishing work zones 105 around the robots 120 and the workers 102. As mentioned above, the work zones 105 can be established with a variety of short range transmission technologies such as, for example, Bluetooth.RTM., near field communication (NFC), or RFID technology. Thus, while discussed herein with respect to RFID, it should be understood that other communications protocols could be used and are contemplated herein. It should also be noted that, while they are generally referred to as "RFID readers," it is understood that RFID readers generally also have the ability to write to RFID tags.
RFID technology tends to have a relatively short range--e.g., on the order of approximately 10 feet. Within this range, RFID tags can be detected, read, and written to by an RFID reader. As a result, this limited range can also be used to establish approximate distances between the RFID tags and the RFID reader. In addition, RFID tags can be both passive and active. A passive RFID tag is powered by electromagnetic induction created when the reader reads the RFID tag. An active RFID tag is powered by a local power source (e.g., the battery for the robot 120) and thus, requires less power to read and can often be read over greater distances, among other things.
As discussed above, in some cases, the RFID tags can also provide location or range information. In this configuration, the inventory control system 200 can establish outer 105a, intermediate 105b, and inner 105c work zones. In this manner, the robot 120 can escalate its evasive action as it gets closer and closer to the worker 102. So, for example, the robot 120 can attempt to divert around the worker 102 upon detection of the outer work zone 105a, slow down upon detection of the intermediate work zone 105b, and stop completely upon detection of the inner work zone 105c.
As mentioned above, the robots 120 can be used to move inventory holders 130 between locations within the warehouse floor 170. The robots 120 may represent many types of devices or components appropriate for use in inventory control system 200 based on the characteristics and configuration of inventory holders 130 and/or other elements of inventory control system 200. In a particular embodiment of inventory control system 200, the robots 120 can represent independent, self-powered devices, such as wheeled or tracked robots or robotic carts, for example, configured to freely move about warehouse floor 170. Examples of such inventory control systems are disclosed in U.S. Patent Publication No. 2012/0143427, published on Jun. 7, 2012, titled "SYSTEM AND METHOD FOR POSITIONING A MOBILE DRIVE UNIT," and U.S. Pat. No. 8,280,547, issued on Oct. 2, 2012, titled "METHOD AND SYSTEM FOR TRANSPORTING INVENTORY ITEMS," the entire disclosures of which are herein incorporated by reference.
In some examples, the warehouse floor 170 floor can also comprise a plurality of markers, or fiducials 175, to enable the robots 120 to establish their location in the warehouse. Because the robots 120 are generally low enough to travel under inventory holders 130 (i.e., to be able to lift them), in some examples, the fiducials 175 can also continue under the inventory holders 130, substantially spanning the entire floor. In some examples, the area between the fiducials 175 can define grid areas 175a with a fiducial 175 at each corner. When attempting to locate a particular inventory holder 130, therefore, the robot 120 can locate the fiducial 175, or grid 175a, associated with the inventory holder's location by scanning the floor with a downward facing scanner or camera and then confirm that it is in the right location by scanning an identifier, e.g., a 2D or 3D bar code, an RFID tag, or other identifier, on the bottom of the inventory holder 130 with an upward facing scanner or camera, for example. In some examples, the inventory holder 130 and/or the fiducials 175 can include 2D or 3D bar codes, an RFID tag, or other identifiers.
Similarly, as shown in FIG. 4, examples of present disclosure can comprise another system 400 for maintaining a predetermined distance between workers 102 and robots using RFID, or other short-range transmission technologies. In some examples, the worker 102 can wear a garment 405 or other wearable device, such as a vest, a hat, or coveralls that includes one or more RFID tags 305. In some examples, the RFID tags 305 can comprise surface RFID tags 305a that can be sewn or adhered, for example, to the surface of the garment 405. In other examples, the RFID tags 305 can be embedded RFID tags 305b that are sewn into, or otherwise incorporated into the garment 405. Embedded RFID tags 305b may improve the aesthetics of the garment 405, for example, or may simply provide some protection to the RFID tags 305 from abrasion or other damage. In this configuration, the RFID reader 310 on the robot 120 constantly polls for RFID tags 305 at a predetermined interval (e.g., once every second, once every 0.5 seconds) that are within range. When the robot 120 receives a return from a RFID tag 305 associated with a worker 102, the robot 120 can take evasive action.
Regardless of the configuration (i.e., FIG. 3 or FIG. 4), in some examples, the systems 300, 400 can also use fiducials 175 in the warehouse floor 170 to establish appropriate work zones 105 based, at least in part, on the range of the RFID readers 310 and tags 305 in the system 300, 400. In other words, in some examples, the fiducials 175 can also include RFID tags 305 to provide location information to RFID readers 310 on the robots 120 and/or workers 102 to establish the range of the RFID readers. When the RFID reader reports to the central control 115, for example, it can include all of the fiducials 175 it can "see" at any given time. Based on location information associated with the reported fiducials 175, therefore, the central control 115 can determine the range of a particular RFID reader 310 or the average range of all RFID readers 310 in the system 300, 400, for example.
The central control 115 can then use this information, in part, to establish the radius used for the outer work zone 105a. The ranges of various RFID components 305, 310 can vary widely based on, for example, atmospheric conditions, local interference, placement, battery charge levels, and angle of incidence between the reader and tag. In addition to RFID range, the size of work zones 105 can also be based on the number of robots 120 and/or workers 102 on the warehouse floor 170, the travel speed of the robots 120, and the size of the warehouse floor 170, among other things. Because some of these variables can change, in some examples, the system 300, 400 can periodically reset work zones 105 based on current conditions.
As shown in FIGS. 5A and 5B, the system 400 can include a vest 505, hat 510, coveralls, belts, or other item of clothing (collectively, "garment") that can be easily worn and can include a plurality of RFID tags 305. In some examples, as shown, the garment can include RFID tags 305 arranged in multiple orientations and locations to enable the RFID tags 305 on the garment to be detected by RFID readers 310 regardless of their relative positions. In other words, the RFID readers 310 can detect at least one tag 305 regardless of whether the worker 102 is walking away or towards the robot 120 and regardless of the worker's relative position to the robot (e.g. in front, behind, or to either side of the robot 120).
In some examples, the garment can comprise RFID tags 305a mounted on the surface. In other embodiments, the garment can comprise RFID tags 305b embedded (e.g., sewn into) the garment. In still other embodiments, the garment can include multiple types of RFID tags 305a, 305b such as, for example, passive, active, and semi-active RFID tags 305.
In other examples, as shown in detail in FIG. 7, the fiducials 175 can also comprise additional fiducial data 705. The fiducial data 705 can comprise, for example, a fiducial identification number (ID) 175b (e.g., "fiducial 186143a"). In some examples, the robot 120 can read the fiducial ID 175b with the camera 605, or other suitable device, and can cross-reference the fiducial ID 175b with an onboard database to establish its location. In other embodiments, the central control 115 can include a fiducial database and the robot 120 can transmit the fiducial ID 175b to the central control 115 and the central control 115 can provide the location of the fiducial 175 to the robot 120.
In other examples, the fiducial data 705 can also comprise, for example, a bar code 175c and/or an RFID tag 305, among other things, that can be read by the robot 120. In some examples, the bar code 175c or RFID tag 305 can have embedded location information to directly provide location information to the robot. In other embodiments, as discussed above, the robot 120 can read or scan the bar code 175c or RFID tag 305, transmit the fiducial data 705 to the central control, and receive location information for the fiducial 175 from the central control 115 to determine the location of the robot 120 on the warehouse floor 170.
It is, of course, possible that the worker 102 could walk away from the cart 615, rendering their location unknown to the central control 115. In other words, when the worker 102 walks away from the cart, the cameras 605 on the cart 615 provide the central control 115 with the location of the cart 615, but not the location of the worker 102. As a result, in some examples, the system 600 can also comprise a "tether" 625 between the cart 615 and the worker 102.
In still other embodiments, the tether 625 can comprise what is essentially an "electromagnetic" tether. In other words, using RFID technology, or other short range link, in a similar manner to that discussed above, the tether 625 can determine whether the worker 102 is within a certain distance of the cart 615. So, for example, as discussed above, the system 600 can include a garment for the worker 102 (e.g., a vest 505) comprising one or more RFID tags 305 and the cart 615 can comprise a RFID reader 310. When the worker 102 is within range of the RFID reader 310, therefore, the tether 625 can be considered "latched," and the system 600 can operate normally.
If the worker 102 is out of range of the RFID reader 310, on the other hand, the tether 625 can be considered "unlatched," send an unlatched signal to the central control 115, which may require the central control 115 to stop all robots 120 in the warehouse, or in a portion of the warehouse, until the worker 102 can be "found." In other words, if the tether 625 is unlatched, the location of the worker 102 on the warehouse floor 170 is essentially unknown. If this is the case, the central control 115 cannot accurately route robots 120 around the worker 102 and may have no choice but to shut down all of the robots 120. In some cases, when the worker 102 returns to the cart 615, the tether 625 can automatically "relatch," send a relatch signal to the central control 115, and normal warehouse operations can resume. In some examples, to prevent a total shutdown, the system 600 can include a secondary location system using, for example, RFID, proximity sensors, infrared cameras, or facial recognition, to locate the worker 102. In a preferred example, workers 102 can simply be trained to stay within range of their cart 615 at all times.
In some examples, a similar system 600 can be used in a work station 150. In other words, while the worker 102 in a work station 150 is theoretically stationary, in some instances, the worker 102 may need to temporarily leave the bounds of the work station 150 to retrieve a dropped item, for example. Thus, while a "permanent" work zone 105 may exist with respect to the work station 150 (to keep robots 120 from driving through the work station 150), if the worker 102 leaves the work station 150, the robots 120 may need to take additional evasive action.
To this end, in addition to the redundancy provided by the use of multiple RFID tags 305 and RFID readers 310, as discussed above, in some examples, the work zones 105, robots 120, and workers 102 can be monitored and managed by a dedicated interaction server 805. In this configuration, the central control 115 can handle the routing and scheduling of robots 120 during normal operation--e.g., retrieving inventory holders 130 and delivering them to work stations--while the interaction server 805 can monitor workers 102 and robots 120 solely for the purpose of preventing incidents. In this manner, the operations of the robots 120 and the workers 102 are not in conflict, reliability is increased via redundant communications and control systems, and downtime and maintenance is reduced by reducing the number of robot 120/worker 102 interactions, among other things.
At 930, if the robot is within the work zone, on the other hand, the system (i.e., the central control or interaction server) can send a command to take evasive action. In some examples, such as with a single layer work zone, the system can simply command the robot to stop. In other examples, the system can take escalating evasive action--e.g., reroute.fwdarw.slow down.fwdarw.stop--as the robot enters an outer, intermediate, and inner work zone, respectively. In this manner, when a robot and a worker are merely traveling close to one another or at an oblique angle, a slight deviation can enable the robot to miss the worker with little interruption to the system. If the worker and the robot are on a collision course, on the other hand, the robot may need to simply stop and yield the right-of-way to the worker. At 935, the system can continue to monitor and control the robots, as necessary, until the worker leaves the warehouse floor.
To this end, as with the mobile robots 120 discussed above, it can be useful to establish one or more work zones 1005, which can include circles 105 (FIG. 10A) or spheres 1005 (FIG. 10B). As before, in some examples, the system 1000 can include an outer work zone 105a, an intermediate work zone 105b, and an inner work zone 105c. This can enable the robotic arm 1020 to take one or more evasive actions as a worker 102 (or robot 120) enters the work zone 105. In some examples, the system 1000 can use multiple spherical work zones 1005
In some examples, the system 1000 may slow down the robotic arm 1020 when a worker 102 enters the outer work zone 105a. The system 1000 may then apply brakes and/or assume a predetermined position when a worker 102 enters the intermediate work zone 105b. Finally, the system 1000 may remove power from the robotic arm 1020 when a worker 102 enters the inner work zone 105c. In some examples, the robotic arm 1020 may put objects down on the floor, or other work surface, and wait for the worker 102 to leave the work zone 105 or take other additional actions during this process.
As with the systems 300, 400, 600 discussed above, the system 1000 can comprise one or more RFID readers/tags and/or imaging devices to maintain a work zone 1005 around the robotic arm. In some examples, as shown in FIG. 10A, the system 1000 can comprise one or more RFID tags 305 disposed on the robotic arm 1020 and an RFID reader 310 on the worker 102 to detect an RFID interaction. In other examples, as shown in FIG. 10B, the system 1000 can utilize multiple surface mount 305a, or embedded 305b, RFID tags on the worker 102 and an RFID reader 1030 on the robotic arm 1020 to detect RFID interactions. In still other examples, the robotic arm 1020 can comprise one or more imaging devices 1025 to provide information related to the position of the robotic arm 1020 and/or work zones 105, 1005.
Previous Patent US 9,889,562 | Next Patent US 9,889,564