Patent Description:
The disclosure herein generally relates to telepresence robots, and, more particularly, to a method and system of dynamic localization of a telepresence robot based on live markers.

In current scenario, creating video conference (Vcon) facilities are expensive and are always fixed setup. Existing systems are not able to follow the entire teleconferencing setup based on location and are not sufficiently intelligent. Also, there are situations where location-based intelligence and awareness is required for the video conference. Interaction with the moving robot may give more flexibility in operation. However, currently available teleconferencing robotic systems are not smart. These also fail to navigate based on specific path and are unable to focus on the presenter based on overall environment.

The document <CIT> describes The present disclosure provides a map generation method, localization method, and simultaneous localization and mapping method. The method includes: recognizing fiducial markers in the motion area; taking a position as origin of a global coordinate system of the robot, and obtaining pose information of the fiducial markers; the robot moving to a next position, recognizing the fiducial markers with determined coordinate information and underdetermined coordinate information respectively, and obtaining pose information of the fiducial marker of the undetermined coordinate information with respect to the origin based on that of the determined coordinate information; repeating the previous step until the pose information of all the fiducial markers are obtained; and generating a marker map associated with coordinate information of all fiducial markers. The method is capable of generating a map of the motion area through the fiducial markers and further realizing autonomous localization.

Embodiments of the present disclosure present technological improvements as solutions to one or more of the above-mentioned technical problems recognized by the inventors in conventional systems. For example, in one aspect, a processor implemented method of dynamic localization of a telepresence robot based on a plurality of live markers is provided. The processor implemented method is embodied in claims <NUM>-<NUM>.

In another aspect, there is provided a system to dynamically localize a telepresence robot based on a plurality of live markers. The system comprises a memory storing instructions; one or more communication interfaces; and one or more hardware processors coupled to the memory via the one or more communication interfaces. The one or more hardware processors are configured by the instructions to: perform steps as embodied in claims <NUM>-<NUM>.

In yet another aspect, there are provided one or more non-transitory machine readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors causes at least one of: perform steps as embodied in claim <NUM>.

<FIG> illustrates a network implementation <NUM> of a system <NUM> for dynamic localization of a telepresence robot based on one or more live markers, according to an embodiment of the present disclosure. The system is adapted to dynamically localize the telepresence robot based on the one or more live markers. Although the present subject matter is explained considering that the system <NUM> is implemented for dynamically localize the telepresence robot based on the one or more live markers and may be understood that the system <NUM> may not be restricted to any specific machine, or environment. In an embodiment, the system <NUM> are implemented in a variety of computing systems, such as laptop computers, notebooks, hand-held devices, workstations, mainframe computers, servers, a network cloud, a smart phone, a wearable device, and the like.

Herein, the system <NUM> may acquire an input data for dynamically localize the telepresence robot based on the one or more live markers via one or more user devices <NUM>-A, <NUM>-B,. <NUM>-N, collectively referred to as user devices <NUM> hereinafter. In an embodiment, the user devices <NUM> may be embodied in handheld electronic device, a mobile phone, a smartphone, a portable computer, a PDA, and so on. The user devices <NUM> are communicatively coupled to the system <NUM> through a network <NUM> and may be capable of providing input data to the system <NUM>.

In one implementation, the network <NUM> may be a wireless network, a wired network or a combination thereof. The network <NUM> can be implemented as one of the different types of networks, such as intranet, local area network (LAN), wide area network (WAN), the internet, and the like. The network <NUM> may either be a dedicated network or a shared network. The shared network represents an association of the different types of networks that use a variety of protocols, for example, Hypertext Transfer Protocol (HTTP), Transmission Control Protocol/Internet Protocol (TCP/IP), Wireless Application Protocol (WAP), and the like, to communicate with one another. Further, the network <NUM> may include a variety of network devices, including routers, bridges, servers, computing devices, storage devices, and the like.

In an embodiment, the system <NUM> may be embodied in the computing device <NUM>. The system <NUM> may also be associated with a data repository <NUM> to store at least data i.e., information associated with the telepresence robot movement, hand eye gesture, content delivery, navigation around an environment from one location to another location using a display marker unit, etc. Additionally, or alternatively, the data repository <NUM> may be configured to store data and/or information associated with one or more live markers. The data repository <NUM> may be configured outside and communicably coupled to the computing device <NUM> embodying the system <NUM>. Alternatively, the data repository <NUM> may be configured within the system <NUM>. An example implementation of the system <NUM> for dynamic localization of the telepresence robot based on one or more live markers described further with reference to <FIG>.

<FIG> illustrates a block diagram of a system <NUM> for dynamic localization of the telepresence robot based on the one or more live markers, according to an embodiment of the present disclosure. The system <NUM> may be an example of the system <NUM> (<FIG>). In an example embodiment, the system <NUM> may be embodied in, or is in direct communication with the system, for example the system <NUM> (<FIG>). In an embodiment, the system <NUM> includes one or more processors <NUM>, communication interface device(s) or input/output (I/O) interface(s) <NUM>, and one or more data storage devices or memory <NUM> operatively coupled to the one or more processors <NUM>. The memory <NUM> comprises a database. The processor <NUM>, memory <NUM>, and the I/O interface <NUM> may be coupled by a system bus such as a system bus <NUM> or a similar mechanism. The one or more processors <NUM> that are hardware processors can be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and/or any devices that manipulate signals based on operational instructions. Among other capabilities, the processor(s) <NUM> is configured to fetch and execute computer-readable instructions stored in the memory <NUM>.

The I/O interface device(s) <NUM> can include a variety of software and hardware interfaces, for example, a web interface, a graphical user interface, and the like. The interfaces <NUM> may include a variety of software and hardware interfaces, for example, interfaces for peripheral device(s), such as a keyboard, a mouse, an external memory, a camera device, and a printer. Further, the interfaces <NUM> may enable the system <NUM> to communicate with other devices, such as web servers and external databases. The interfaces <NUM> can facilitate multiple communications within a wide variety of networks and protocol types, including wired networks, for example, local area network (LAN), cable, etc., and wireless networks, such as Wireless LAN (WLAN), cellular, or satellite. In an embodiment, the I/O interface device(s) can include one or more ports for connecting number of devices to one another or to another server.

The memory <NUM> may include any computer-readable medium known in the art including, for example, volatile memory, such as static random-access memory (SRAM) and dynamic random-access memory (DRAM), and/or non-volatile memory, such as read only memory (ROM), erasable programmable ROM, flash memories, hard disks, optical disks, and magnetic tapes. In an embodiment, the memory <NUM> includes a plurality of modules <NUM> and a repository <NUM> for storing data processed, received, and generated by the plurality of modules <NUM>. The plurality of modules <NUM> may include routines, programs, objects, components, data structures, and so on, which perform particular tasks or implement particular abstract data types.

Further, the database stores information pertaining to inputs fed to the system and/or outputs generated by the system (e.g., data/output generated at each stage of the data processing), specific to the methodology described herein. More specifically, the database stores information being processed at each step of the proposed methodology.

Additionally, the other modules <NUM> may include programs or coded instructions that supplement applications and functions of the system <NUM>. The repository <NUM>, amongst other things, includes a system database <NUM> and other data <NUM>. The other data <NUM> may include data generated as a result of the execution of one or more modules in the module (s) <NUM>. Herein, the memory for example the memory <NUM> and the computer program code configured to, with the hardware processor for example the processor <NUM>, causes the system <NUM> to perform various functions described herein under.

<FIG> illustrates an exemplary architecture block diagram of the system for dynamic localization of the telepresence robot based on the one or more live markers, according to an embodiment of the present disclosure. The system <NUM> is divided in to plug and play modules and each module is implemented in a way that adheres to a standard framework for communication and execution. At the bottom, a robot operating system (R. S) framework layer is a configured for abstracting actual hardware connected to the system <NUM> and optimally sharing one or more hardware between one or more top nodes. In an embodiment, the one or more hardware related interfacing/data transfer required by one or more system applications running on the telepresence robot are routed through the robot operating system (R. S) framework layer.

An application driver layer is above the ROS layer which is an abstraction layer provide the above layer with one or more libraries, an application programming interface (API), hardware capabilities like a motion of the telepresence robot, a voice interface, a vision system, and other cloud-based interface libraries. The application driver layer enables one or more application developers with one or more tools and a documentation for building one or more software applications on the telepresence robot using the hardware capabilities, cloud-based capabilities like a chat interaction API, an image processing API, and also an autonomous navigation and an API's for path planning of the telepresence robot. An application layer is topmost layer, in which one or more final applications are running. In an embodiment, one or more users' applications runs at the application layer. The one or more applications are configured to orchestrate one or more functionalities of the at least one hardware such as a robot movement, a hand eye gesture, a content delivery, a navigation around an environment using the display marker unit, etc. The system architecture is developed in a way that ensures the system <NUM> can be used as a platform for customer specific requirements. Modules can be independently switched in the application driver layer, to provide optimal hardware utilization with better overall performance. The application layer uses, a mobile hybrid application development platform, which opens a window to a whole new generation of telepresence robots where one can develop hybrid apps to be deployed on the telepresence robot.

<FIG> illustrates an exemplary block diagram of a display marker unit, according to an embodiment of the present disclosure. <FIG> illustrates an exemplary block diagram of components of the display marker unit, according to an embodiment of the present disclosure. The display marker unit include a dedicated processing unit e.g., microcontroller <NUM>, a battery <NUM>, a wireless communication hardware e.g., Wi-Fi wireless <NUM>, and an electronic paper display <NUM>. For example, the display marker unit with certain content on a display. The contents on the marker change dynamically based on a trajectory of the telepresence robot or next goal (e.g., another location) to be reached by the telepresence robot. In an embodiment, at least one live marker changes at least one content to orient the telepresence robot into a required trajectory. The information associated with a change in orientation of the telepresence robot is communicated to at least one neighboring marker to dynamically change associated marker contents to further navigate the telepresence robot in a right direction. In an embodiment, the at least one neighboring marker is same as of the at least one live marker to control one or more activities of the telepresence robot. In an embodiment, the associated marker contents correspond to information regarding the path, activity mapping, and information to process on next location. For example, the associated marker contents are communicated though number of alphanumeric information which is decoded with a lookup table from a cloud. For example, using a minimum of three display marker units the location of the telepresence robot can be identified and a trajectory can be planned in a way after moving the telepresence robot a certain distance, a marker comes in the robot's view.

The telepresence robot is moved based on information associated with the one or more live markers. The telepresence robot is a differential robot that navigates based on the difference in one or more velocities between two drive wheels. A velocity of a right wheel and a velocity of a left wheel of the telepresence robot is given by, <MAT> <MAT>.

If Vleft = Vright, the telepresence robot moves in a straight line. Based on the difference between the one or more velocities of the left wheel and the right wheel the telepresence robot respectively which can move in any direction.

If the telepresence robot does not see any display marker unit, the telepresence robot can request the display marker unit to triangulate the location of the telepresence robot, using the RSSI (received signal strength indicator) value for the wireless link between the one or more live markers and the telepresence robot.

<FIG> is an exemplary graphical representation illustrating how three display marker units compute a location of the telepresence robot, according to an embodiment of the present disclosure. For example, at least three display marker units are required to compute location of the telepresence robot based on trilateration. (x1, y1), (x2, y2), (x3, y3) are location of the three display marker units. d<NUM>,d<NUM>,d<NUM> are the distance between each display marker unit and the telepresence robot which is calculated based on the RSSI values.

The location of the telepresence robot (x, y) is obtained from the below three equations. <MAT> <MAT> <MAT>.

<FIG> is an exemplary flow diagram illustrating a method of dynamically localizing the telepresence robot based on the one or more live markers, according to an embodiment of the present disclosure. In an embodiment, the system <NUM> comprises one or more data storage devices or the memory <NUM> operatively coupled to the one or more hardware processors <NUM> and is configured to store instructions for execution of steps of the method by the one or more processors <NUM>. The flow diagram depicted is better understood by way of following explanation/description. The steps of the method of the present disclosure will now be explained with reference to the components of the system as depicted in <FIG>.

At step <NUM>, one or more images are received from an image capturing device connected to the telepresence robot. At step <NUM>, the one or more images are processed to identify the one or more live markers in a path of the telepresence robot. In an embodiment, the one or more live markers are at least one marker whose display code can be changed dynamically. For example, in a screen a quick response (QR) code is displayed for a web link and later reflect to some other content after some period. The one or more live markers are showed through a led screen and driven through a controller. The controller can be given information through a Bluetooth® or Wi-Fi based communication either by the one or more users or automatically. In an embodiment, the path is a motion area of the telepresence robot, where the camera device of the telepresence robot monitors the one or more live markers while navigation of the telepresence robot. At step <NUM>, a binary matrix is decoded to identify at least one identifier (ID) associated with the at least one live marker from the one or more live markers. In an embodiment, the binary matrix is stored in the memory <NUM> but not limited to a cloud. For example, an information is read in any form from the code of the binary matrix. The binary matrix corresponds to a location of the at least one live marker in the static map. In an embodiment, the static map is a usual path map for navigation of the telepresence robot. The telepresence robot is already fed with multiple maps and a map related to the information conceived from the at least one live marker is computed from the existing information to navigate to a correct location.

At step <NUM>, one or more parameters are identified based on the at least one identifier (ID) associated with the at least one live marker. The one or more parameters corresponds to at least one: (a) a size, (b) a location, or (c) an orientation. In an embodiment, the one or more parameters are identified based on content on the at least one live marker e.g., like process of reading the QR code. For example, the content of the at least one live marker includes one or more values which is later mapped to the look up table to understand the values. At step <NUM>, a further path to navigate the telepresence robot is dynamically localized based on the one or more parameters and the one or more live markers. In an embodiment, the one or more parameters read from the at least one live marker is utilized to decide what is a next information that needs to be showcased to the one or more users. For example, when the telepresence robot is on a path to deliver a food to a room in a hotel and while moving back to a corresponding docking station, which sees a live marker on the path to navigate to another room to collect a bag who is checking out. Further, a receptionist instead of communicating the same directly to the telepresence robot with all the information is queued in a live QR code displayer to execute one by one based on priority by the telepresence robot. Hence, the telepresence robot observes the live marker to decide to enable the travel path.

The one or more orientations and position of the one or more live markers received in the telepresence robot by the electronic paper display <NUM> of the display marker unit. In an embodiment, at least one live marker changes at least one content to orient the telepresence robot into a required trajectory. In an embodiment, the information associated with the change in orientation of the telepresence robot is communicated to the at least one neighboring marker to dynamically change associated marker contents to further navigate the telepresence robot in a right direction. The path of the robot is always trained and if any change from an existing path is identified in a central server, and automatically notify an administrator and a manual intervention is performed. In an embodiment, the associated marker contents correspond to information regarding the path, activity mapping, and information to process on next location. For example, the associated marker contents are communicated though a number of alphanumeric info which is decoded with a lookup table from the cloud).

In an embodiment, a request is further communicated to the display marker unit to triangulate the location of the telepresence robot, using a received signal strength indicator value (RSSI) for a wireless link between the one or more live markers and the telepresence robot. In an embodiment, a failure detection in the display marker unit enables the one or more neighboring markers to redirect a motion plan and a flow of content at the telepresence robot to an alternate path computed based on one or more constraints. For example, if an information in at least one display marker unit is not aligned with other display marker unit, a failure is detected. The at least one display marker unit includes a previous command and a current command embedded into the at least one live marker. In an embodiment, the content corresponds to an action that the telepresence robot should perform based on the failure or change in command. In an embodiment, the one or more constraints corresponds to at least one of (a) an interaction results, and (b) an optimal distance to be travelled between the display marker units. In an embodiment, the interaction results correspond to missing of information of any live marker or misinterpretation of information of value of the at least one live marker. The telepresence robot knows distance of each display marker unit and base interaction guidelines as one or more commands are driven.

In an exemplary embodiment, the one or more dynamically marker units are connected via a wireless network to the administrator and the telepresence robot. The wireless network enables communications as well as a proximate location triangulation of the telepresence robot, if the display marker units are not in a right direction. Based on the input from the administrator, regarding a language of a content delivery and a content to be delivered by the telepresence robot, the one or more markers in the environment changes a binary matrix which convey necessary information to the telepresence robot. In an embodiment, the binary matrix corresponds to at least one of (a) markers barcodes, (b) aruco markers, and (c) Apriltags etc. In an embodiment, based on the input from the administrator, the telepresence robot delivers the content in specific language and in a specific order and simultaneously interacts with the people. If the telepresence robot learns from an interaction that a target audience is interested in very specific content or in a specific language, the telepresence robot generates one or more suggestions to the administrator regarding an update of a way the content needs to be delivered. Based on approval from the administrator's, a new language, the content, and the navigation order for the telepresence robot are updated by the display marker unit. In an embodiment, certain low-level changes, navigation changes or the delivery content changes, are dynamically managed by the wireless network of one or more display units and the telepresence robot.

For example, a presentation demo scenario where the telepresence robot is initially situated near the entrance. When a guest arrives near the telepresence robot and interacts, the telepresence robot checks for markers around (in this case i.e., Marker A) and simultaneously updates about the guest and the request query to the central server. Based on which the content on the marker A is updated by the server and the telepresence robot delivers the content in a particular flow. After content for initial place is over according to the initial request marker A redirects the telepresence robot to marker B for next content. Meanwhile, in case any other query from the guest, the server can change the content on the markers to change flow of conversation. So, in some other scenario, according to the query, the telepresence robot might be redirected from marker A to marker B directly. Example scenario like shopping malls where markers are placed for every row of shelves with items. When a user interacts with the telepresence robot regarding a query for some item and the user say that the shelf is empty, the feedback is communicated to the administrator for reconciliation of items and the neighboring markers update the information redirecting upcoming user through a proper conversation and suggesting them to either get an alternative or to notify the person in-charge to support the user.

The embodiments of present disclosure herein provide the telepresence robot which can interact with the one or more users and provide a video conferencing environment in which any user can control, view and interact with other users. The telepresence robot can be controlled from anywhere remotely. The one or more users of the telepresence robot can initiate commands in a voice mode to control. The telepresence robot provides an autonomous navigation and intelligent localization-based event management. The event management corresponds to activity to be performed by the telepresence robot at a particular location. For example, the activity may be "To talk about quantum computing with some examples at one point" and similarly "To perform an OCR based customer case study discussion at other point". The telepresence robot includes an intelligent localization capability to detect a location with at least one live marker and associated information, and present a specific content or follow a presenter. The telepresence robot would be able to do a presentation using an inbuilt memory stored data or from the cloud. The telepresence robot can follow a person using an image and face detection algorithm.

The embodiments of the present disclosure herein include a location awareness feature which is enabled to move inside a closed area and identify at least one specific zone while in a presentation mode. The location awareness is through a location of the at least one live marker and a global positioning system (GPS). Also, using the image processing techniques, a position is adjusted based on a specific zone by looking at a presentation artifact. The telepresence robot includes an ability to provide information about a zone and start with enabling the presentation broadcasting or provide associated with a presentation data based on the feed. Based on type of the event, the telepresence robot includes a capability to pre-map area and provide only required information for the presentation happening for the period. Each zone includes a unique identifier to understand exact location for the at least one zone. For example, if the telepresence robot is in a covid ward in the hospital which exactly knows a protocol to be followed and a delivery information, whereas in non-covid wards follows a different protocol. The telepresence robot can enable itself and intelligently deliver the presentation for the event. The telepresence robot can be controlled by anyone with proper authentication credentials and the manual mode provide an interactive feature of the telepresence robot enabled. The telepresence robot is voice enabled and so can intelligently answer any queries.

The telepresence robot can be used in conference meetings to represent an attendee who could not be part of a meeting and can be controlled by the person at another location in a remote manner. A content which is to be articulated by the attendee can be fed to the telepresence robot, which speaks out as required. The telepresence robot also be able to act as a dynamic presentation generator based on one or more queries that are being posed over a period. Also, the telepresence robot can be used to showcase presentations that are placed in places like business centers.

Claim 1:
A processor implemented method (<NUM>), comprising:
receiving (<NUM>), via one or more hardware processors (<NUM>), a plurality of images from an image capturing device connected to a telepresence robot;
processing (<NUM>), via the one or more hardware processors (<NUM>), the plurality of images to identify a plurality of live markers in a path of the telepresence robot, wherein
the plurality of live markers is displayed via a Light Emitting Diode (LED) screen,
the plurality of live markers includes a display code associated with content,
the display code of the plurality of markers changes dynamically,
at least one live marker of the plurality of live markers changes the content to orient the telepresence robot into a required trajectory,
information associated with a change in the orientation of the telepresence robot is communicated to at least one neighboring live marker of the plurality of live markers to dynamically change specific content of the at least one neighboring live marker, to further navigate the telepresence robot in a desired direction,
the specific content is specific information to process on specific location, and
the content corresponds to at least one of: (a) information regarding the path, (b) activity mapping, and (c) information to process on next location;
decoding(<NUM>), via the one or more hardware processors (<NUM>), a binary matrix to identify at least one identifier (ID) associated with the at least one live marker of the plurality of live markers ;
identifying (<NUM>), via the one or more hardware processors (<NUM>), a plurality of parameters based on the at least one identifier (ID) associated with the at least one live marker, wherein the plurality of parameters corresponds to at least one of: (a) a size, (b) a location, or (c) an orientation ; and
dynamically localizing (<NUM>), via the one or more hardware processors(<NUM>) , a further path to navigate the telepresence robot based on the plurality of parameters and the plurality of live markers .