Patent Description:
In recent years, a technology for a robot that is disposed in an indoor space to provide services to a user has been actively developed. Particularly, the robot for cleaning an indoor space may identify surrounding objects through a plurality of sensors and provide a service based on information on the identified object. For example, <CIT> discloses a camera calibration which can be performed in a mobile environment. One or more cameras can be mounted on a mobile vehicle, such as an unmanned aerial vehicle (UAV) or an automobile. Because of the mobility of the vehicle the one or more cameras may be subjected to inaccuracy in imagery caused by various factors, such as environmental factors, impact by other objects, vehicle vibrations, and the like. <CIT> describes operations which comprise capturing, at a vehicle as the vehicle travels, LIDAR scans and camera images. The operations may further comprise selecting, at the vehicle as the vehicle travels, a subset of the LIDAR scans and the camera images that are determined to be useful for calibration. <CIT> relates to calibrating sensor arrays, including sensors arrays mounted on an autonomous vehicle. Image data from multiple cameras in the sensor array can be projected into other camera spaces using one or more dense depth maps. The dense depth map(s) can be generated from point cloud data generated by one of the sensors in the array. Differences determined by the comparison can indicate alignment errors between the cameras.

However, the plurality of sensors provided in the robot may be deteriorated in its performance due to mechanical distortion due to an external physical impact or superannuating. The robot of the related art does not have a function of calibrating the performance, even in a case where the performance of the sensor is deteriorated, and accordingly, a decrease in quality of the service provided by the robot may not be prevented.

Therefore, there is constant demands for the robot that continuously performs calibration for at least one sensor of a plurality of sensors and provides high-quality services.

The invention is made in view of the above needs and an object thereof is to provide a robot that moves to a predetermined point after a predetermined event has occurred, and performs calibration for at least one sensor of a plurality of sensors, and a controlling method thereof.

According to the invention, there is provided a robot including a plurality of sensors, a memory, a driving unit, and a processor configured to, based on identifying that a predetermined event has occurred, control the driving unit so that the robot moves to a predetermined point corresponding to a reference image, based on identifying that the robot has moved to the predetermined point, obtain a plurality of images through the plurality of sensors, identify whether it is necessary to perform calibration for at least one sensor of the plurality of sensors based on the reference image and the plurality of obtained images, based on identifying that it is necessary to perform the calibration for the at least one sensor, obtain calibration data for calibrating sensing data corresponding to the at least one sensor based on a synthesis of the plurality of images and store the obtained calibration data in the memory, based on the sensing data being obtained from the at least one sensor, calibrate the obtained sensing data based on the calibration data stored in the memory, and control the driving unit based on the calibrated sensing data, wherein the memory is configured to store the reference image related to the predetermined point, and wherein the processor is further configured to: obtain a depth image based on a synthesis of the plurality of obtained images, compare the reference image with the obtained depth image, and identify whether it is necessary to perform the calibration for the at least one sensor of the plurality of sensors based on a result of the comparison.

The processor may be configured to identify whether there is a mechanical distortion on each of the plurality of sensors based on the plurality of obtained images, and based on identifying that there is the mechanical distortion on the at least one sensor of the plurality of sensors, identify that it is necessary to perform calibration for the at least one sensor.

The processor may be configured to, based on identifying that a predetermined time interval or an interval in which a predetermined number of tasks has been performed has arrived, identify that the predetermined event has occurred.

The processor may be configured to, based on identifying that the robot has docked at a charge station, identify that the robot has moved to the predetermined point.

The robot may further include a distance sensor, and the processor may be configured to identify whether a dynamic object exists in a surrounding environment of the predetermined point based on sensing data obtained by the distance sensor, after identifying that the robot has moved to the predetermined point, and based on identifying that the dynamic object exists, finish a calibration operation for the plurality of sensors.

The robot may further include a user interface, and a communication interface, and the processor may be configured to, based on a user command being received through at least one of the user interface or the communication interface, control the driving unit so that the robot moves to the predetermined point.

The processor may be configured to, based on the calibration data being obtained, additionally obtain sensing data from the at least one sensor, obtain calibrated sensing data by applying the calibration data to the additionally obtained sensing data, and based on identifying that the calibrated sensing data is improved than the sensing data by a threshold value or more, store the obtained calibration data in the memory.

The processor may be configured to, based on an event in which a traveling mode of the robot is changed occurring after identifying that the robot has moved to the predetermined point, finish a calibration operation for the plurality of sensors.

According to another aspect of the invention, there is provided a system including the robot according to the invnetion including a plurality of sensors, and a user terminal, the system including the user terminal configured to, based on a user command for performing calibration for the sensors provided in the robot being input, transmit the user command to the robot,.

According to another aspect of the invention, there is provided a method for controlling a robot, the method including, based on identifying that a predetermined event has occurred, moving the robot to a predetermined point corresponding to a reference image, based on identifying that the robot has moved to the predetermined point, obtaining a plurality of images through a plurality of sensors, identifying whether it is necessary to perform calibration for at least one sensor of the plurality of sensors based on the reference image and the plurality of obtained images, based on identifying that it is necessary to perform the calibration for the at least one sensor, obtaining and storing calibration data for calibrating sensing data corresponding to the at least one sensor based on a synthesis of the plurality of images, based on the sensing data being obtained from the at least one sensor, calibrating the obtained sensing data based on the stored calibration data, and driving the robot based on the calibrated sensing data, wherein the identifying of whether it is necessary to perform the calibration comprises: obtaining a depth image based on the plurality of obtained images; comparing a reference image related to the predetermined point with the obtained depth image; and identifying whether it is necessary to perform the calibration for the at least one sensor of the plurality of sensors (<NUM>) based on a comparison result.

The identifying whether it is necessary to perform the calibration may include identifying whether there is a mechanical distortion on at least one of the plurality of sensors based on the plurality of obtained images, and based on identifying that there is the mechanical distortion on the at least one sensor of the plurality of sensors, identifying that it is necessary to perform the calibration for the at least one sensor.

The moving the robot may include, based on identifying that at least one of a predetermined time interval or an interval in which a predetermined number of tasks has been performed has arrived, identifying that the predetermined event has occurred.

The obtaining the plurality of images may include, based on identifying that the robot has docked at a charge station, identifying that the robot has moved to the predetermined point.

According to various embodiments of the invention, the robot may obtain sensing data with high reliability and provide a service based thereon, and accordingly, thereby improving user's convenience.

Hereinafter, the invention will be described in detail with reference to the accompanying drawings.

The terms used in embodiments of the invention have been selected as widely used general terms as possible in consideration of functions in the invention, but these may vary in accordance with the intention of those skilled in the art, the precedent, the emergence of new technologies and the like. In addition, in a certain case, there may also be an arbitrarily selected term, in which case the meaning will be described in the description of the invention. Therefore, the terms used in the invention should be defined based on the meanings of the terms themselves and the contents throughout the invention, rather than the simple names of the terms.

In this invention, the terms such as "comprise", "may comprise", "consist of", or "may consist of" are used herein to designate a presence of corresponding features (e.g., constituent elements such as number, function, operation, or part), and not to preclude a presence of additional features.

It should be understood that the expression such as "at least one of A or/and B" expresses any one of "A", "B", or "at least one of A and B".

The expressions "first," "second" and the like used in the invention may denote various elements, regardless of order and/or importance, and may be used to distinguish one element from another, and does not limit the elements.

If it is described that a certain element (e.g., first element) is "operatively or communicatively coupled with/to" or is "connected to" another element (e.g., second element), it should be understood that the certain element may be connected to the other element directly or through still another element (e.g., third element).

Unless otherwise defined specifically, a singular expression may encompass a plural expression. It is to be understood that the terms such as "comprise" or "consist of" are used herein to designate a presence of characteristic, number, step, operation, element, part, or a combination thereof, and not to preclude a presence or a possibility of adding one or more of other characteristics, numbers, steps, operations, elements, parts or a combination thereof.

A term such as "module" or a "unit" in the invention may perform at least one function or operation, and may be implemented as hardware, software, or a combination of hardware and software. Further, except for when each of a plurality of "modules", "units", and the like needs to be realized in an individual hardware, the components may be integrated in at least one module and be implemented in at least one processor (not illustrated).

A "user" in the invention may refer to a person who receives a service from the robot but is not limited thereto.

<FIG> is a diagram illustrating an operation of the robot that travels an indoor space.

According to <FIG>, a robot <NUM> according to an embodiment of the invention may travel a specific space and provide a service to a user. For example, the robot <NUM> may provide a service of cleaning a space but is not limited thereto. The robot <NUM> may provide the service based on data obtained through a sensor <NUM> and the sensor <NUM> according to an example may be an optical sensor.

The sensor <NUM> according to an example may include a plurality of cameras. Since the camera may be deteriorated in its performance due to the mechanical distortion due to an external physical impact and superannuating of the robot <NUM>, the robot <NUM> may perform calibration for the sensor <NUM>. For example, if it is identified that there is a change in an angle of view of at least one camera among a left camera and a right camera included in the robot <NUM>, the calibration for the corresponding camera may be performed.

The robot <NUM> may perform the calibration for the sensor <NUM> in a state of docking at a charge station <NUM>. Herein, the charge station <NUM> may include a device that supplies power for charging the robot <NUM> when it is identified that the robot <NUM> has docked at the charge station <NUM>, but is not limited thereto.

In addition, the robot <NUM> may obtain calibration data based on an image obtained through at least one camera identified as requiring the calibration, and provide a service based on calibrated sensing data obtained by applying the calibration data to sensing data obtained through the sensor <NUM>.

<FIG> is a diagram illustrating mechanical distortion of the sensor provided in the robot and the calibration operation thereof. Referring to <FIG>, the calibration for the sensor <NUM> will be described by assuming that the sensor <NUM> is a camera module including a plurality of cameras.

The sensor <NUM> according to an example may be implemented in a form that two cameras are disposed on a substrate, and the robot <NUM> may determine whether there is mechanical distortion on any one camera of the two cameras.

Specifically, the robot <NUM> may determine whether there is mechanical distortion on the remaining camera based on a predetermined camera or a randomly selected camera. Herein, the mechanical distortion may occur due to physical impact applied to the sensor <NUM>, and the mechanical distortion may include various patterns, for example, x axis direction shift <NUM>, y axis direction shift <NUM>, z axis direction shift <NUM>, x axis center rotation <NUM>, y axis center rotation <NUM>, and z axis center rotation <NUM>, but is not limited thereto.

Referring to <FIG>, the robot <NUM> may determine whether there is the mechanical distortion on the right camera based on the left camera. In addition, the robot <NUM> may store threshold errors corresponding to various patterns included in the mechanical distortion, and the robot <NUM> may determine whether there is a pattern having an error equal to or more than the threshold error among the various patterns included in the mechanical distortion on the right camera based on two images obtained through the two cameras.

In addition, the robot <NUM> may store calibration related information corresponding to the various patterns described above, and if it is identified that the right camera has the mechanical distortion, the robot <NUM> may calibrate the pattern of the mechanical distortion occurred on the right camera based on the calibration related information in terms of software.

Hereinafter, various embodiments of moving to a predetermined point and performing the calibration for at least one sensor of a plurality of sensors after occurrence of a predetermined calibration event will be described in more detail.

<FIG> is a diagram illustrating a configuration of the robot according to an embodiment.

Referring to <FIG>, the robot <NUM> may include a plurality of sensors <NUM>, a memory <NUM>, a driving unit <NUM>, and a processor <NUM>.

The plurality of sensors <NUM> may include a first sensor <NUM>-<NUM> to an n-th sensor <NUM>-n. Each of the first sensor <NUM>-<NUM> to the n-th sensor <NUM>-n may measure physical quantity or detect an operation state of the robot <NUM> and convert the measured or detected information into an electrical signal. The plurality of sensors <NUM> may include a camera, and the camera may include a lens which focuses visible light received due to reflection by an object and other optical signals by an image sensor, and the image sensor capable of detecting the visible light and the other optical signals. Herein, the image sensor may include a 2D pixel array divided into a plurality of pixels and the camera according to an example may be implemented as a depth camera.

In addition, the plurality of sensors <NUM> may include at least one of a microphone, a distance sensor, a gesture sensor, a gyro sensor, a pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor (e.g., RGB (red, green blue) sensor), a biological sensor, a temperature/humidity sensor, a luminance sensor, or an ultra violet (UV) sensor, in addition to the camera.

The memory <NUM> may store data necessary for various embodiments of the invention. The memory <NUM> may be implemented in a form of a memory embedded in the robot <NUM> or implemented in a form of a memory detachable from the robot <NUM> according to data storage purpose. For example, data for driving the robot <NUM> may be stored in a memory embedded in the robot <NUM>, and data for an extended function of the robot <NUM> may be stored in a memory detachable from the robot <NUM>. Meanwhile, the memory embedded in the robot <NUM> may be implemented as at least one of a volatile memory (e.g., a dynamic RAM (DRAM), a static RAM (SRAM), a synchronous dynamic RAM (SDRAM), or the like), a non-volatile memory (e.g., one time programmable ROM (OTPROM), a programmable ROM (PROM), an erasable and programmable ROM (EPROM), an electrically erasable and programmable ROM (EEPROM), a mask ROM, a flash ROM, a flash memory (e.g., a NAND flash or a NOR flash), a hard drive or a solid state drive (SSD), and the like. In addition, the memory detachable from the robot <NUM> may be implemented as a memory card (e.g., a compact flash (CF), secure digital (SD), a micro secure digital (Micro-SD), a mini secure digital (Mini-SD), extreme digital (xD), a multi-media card (MMC), or the like), an external memory connectable to a USB port (e.g., a USB memory), and the like.

The driving unit <NUM> may be a device capable of allowing the robot <NUM> to travel. The driving unit <NUM> may adjust a traveling direction and a traveling speed according to the control of the processor <NUM> and the driving unit <NUM> according to an example may include a power generator for generating power for the robot <NUM> to travel (e.g., a gasoline engine, a diesel engine, a liquefied petroleum gas (LPG) engine, an electrical motor, and the like depending on use of fuel (or energy source)), a steering for adjusting the traveling direction (e.g., manual steering, a hydraulics steering, an electronic control power steering (EPS), and the like), a traveler for allowing the robot <NUM> to travel according to the power (e.g., a wheel, a propeller, and the like), and the like. Herein, the driving unit <NUM> may be modified depending on the traveling type (e.g., wheel type, walking type, or flying type) of the robot <NUM>.

The processor <NUM> generally controls operations of the robot <NUM>. Specifically, the processor <NUM> may be connected to each constituent element of the robot <NUM> to generally control the operations of the robot <NUM>. For example, the processor <NUM> may be connected to the plurality of sensors <NUM>, the memory <NUM>, and the driving unit <NUM> to control the operations of the robot <NUM>.

According to an embodiment, the processor <NUM> may be referred to various names such as a digital signal processor (DSP), a microprocessor, a central processing unit (CPU), a Micro Controller Unit (MCU), a micro processing unit (MPU), a Neural Processing Unit (NPU), a controller, an application processor (AP), and the like, but the processor <NUM> is used in the invention.

The processor <NUM> may be implemented as System on Chip (SoC) or large scale integration (LSI) or may be implemented in form of a field programmable gate array (FPGA). In addition, the processor <NUM> may include a volatile memory such as an SRAM.

When it is identified that a predetermined event has occurred, the processor <NUM> according to an embodiment of the invention may control the driving unit <NUM> so that the robot <NUM> moves to a predetermined point. Herein, the predetermined event may include at least one of an event in which a predetermined calibration interval has arrived after the calibration for the plurality of sensors <NUM> is performed finally or an event in which a user command for performing the calibration for the plurality of sensors <NUM> is input, but is not limited thereto.

For example, if it is identified that at least one of a predetermined time interval or an interval in which a predetermined number of tasks has been performed has arrived, the processor <NUM> may identify that the predetermined event has occurred. For example, if it is identified that the robot <NUM> performs a task <NUM> times without performing the calibration for the plurality of sensors <NUM> or one week has elapsed from the point when the calibration for the plurality of sensors <NUM> is performed finally, the processor <NUM> may identify that the predetermined event has occurred and control the driving unit <NUM> so that the robot <NUM> moves to the predetermined point.

In addition, if it is identified that the robot <NUM> has moved to the predetermined point, the processor <NUM> may obtain a plurality of images through the plurality of sensors <NUM>. For example, if it is identified that the robot <NUM> is located within a threshold distance from the charge station <NUM> based on map data stored in the memory <NUM> or it is identified that the robot <NUM> has docked at the charge station <NUM>, it is identified that the robot <NUM> has moved to the predetermined point, and the robot may obtain the plurality of images through the plurality of sensors <NUM>.

In addition, the processor <NUM> may identify whether the calibration is necessary for at least one sensor of the plurality of sensors <NUM> based on the plurality of obtained images. For example, the processor <NUM> may identify whether there is the mechanical distortion on each of the plurality of sensors <NUM> based on the plurality of images obtained through the plurality of sensors <NUM>, and if it is identified that there is the mechanical distortion on at least one sensor, the processor may identify that it is necessary to perform the calibration for the at least one sensor.

For example, the memory <NUM> may store a reference image related to the predetermined point, and the processor <NUM> may obtain a depth image based on the plurality of obtained images, compare the reference image stored in the memory <NUM> with the obtained depth image, and identify that it is necessary to perform the calibration for at least one sensor of the plurality of sensors <NUM> based on the comparison result. For example, if a similarity between the reference image and the depth image is equal to or less than a threshold value, the processor <NUM> may identify that it is necessary to perform the calibration for the at least one sensor.

In addition, if it is identified that it is necessary to perform the calibration for the at least one sensor, the processor <NUM> may obtain calibration data for calibrating sensing data corresponding to the at least one sensor based on the plurality of images obtained through the plurality of sensors <NUM> and store the calibration data in the memory <NUM>. Herein, the sensing data may include images obtained by the sensor after obtaining the calibration data, but is not limited thereto.

For example, if a similarity between the depth image obtained by synthesizing the plurality of images obtained by the plurality of sensors <NUM> while the robot <NUM> is located at the predetermined point and the reference image related to the predetermined point stored in the memory <NUM> is equal to or less than the threshold value, the processor <NUM> may obtain calibration data for calibrating the image obtained through the camera in terms of software and store this in the memory <NUM>.

If first calibration data pre-stored in the memory <NUM> exists as a result of performing the calibration for the plurality of sensors <NUM> by the robot <NUM> previously, the processor <NUM> may replace (update) the pre-stored first calibration data with second calibration data obtained as a result of recent calibration.

Herein, when the calibration data is obtained, the processor <NUM> may additionally obtain sensing data from at least one sensor, apply the obtained calibration data to the additionally obtained sensing data, and obtain the calibrated sensing data. The obtaining and storing the calibration data will be described below in detail with reference to <FIG> and <FIG>.

In addition, if it is identified that the calibrated sensing data is improved than the sensing data before the calibration by a threshold value, the processor <NUM> may store the obtained calibration data in the memory <NUM>. This will be described below with reference to <FIG>.

In addition, when the sensing data is obtained from at least one sensor, the processor <NUM> may calibrate the obtained sensing data based on the calibration data stored in the memory <NUM> and control the driving unit <NUM> to provide the service based on the calibrated sensing data.

Further, the robot <NUM> may further include a distance sensor, and the processor <NUM> may identify that the robot <NUM> has moved to the predetermined point and then identify whether a dynamic object exists in a surrounding environment of the predetermined point based on the sensing data obtained by the distance sensor. If it is identified that the dynamic object exists in the surrounding environment, the processor <NUM> may end the calibration operation for the plurality of sensors <NUM>.

In addition, the robot <NUM> may further include a user interface and a communication interface, and if a user command is received through at least one of the user interface or the communication interface, the processor <NUM> may control the driving unit <NUM> so that the robot <NUM> moves to the predetermined point.

In addition, if an event in which a traveling mode of the robot <NUM> has occurred after identifying that the robot <NUM> has moved to the predetermined point, the processor <NUM> may end the calibration operation for the plurality of sensors <NUM>.

<FIG> is a diagram specifically illustrating a functional configuration of the robot according to an embodiment of the invention.

According to <FIG>, a robot <NUM>' may include a camera <NUM>-<NUM>, the memory <NUM>, the driving unit <NUM>, the processor <NUM>, a distance sensor <NUM>, a user interface <NUM>, and a communication interface <NUM>. The detailed description of the constituent elements of <FIG> which are overlapped with the constituent elements of <FIG> will not be repeated.

The camera <NUM>-<NUM> may be implemented in a form of a camera module including a plurality of lenses and a plurality of image sensors. According to an example, the camera <NUM>-<NUM> may be a stereo camera including two lenses and two image sensors but is not limited thereto. The processor <NUM> may identify whether there is the mechanical distortion on the camera <NUM>-<NUM>, and if it is identified that there is the mechanical distortion on the camera <NUM>-<NUM> and it is necessary to calibrate this, the processor may obtain calibration data for calibrating an image obtained by the camera <NUM>-<NUM> and store the calibration data in the memory <NUM>.

The distance sensor <NUM> may obtain distance data. Specifically, the distance sensor <NUM> may measure a distance between a location of the robot <NUM> and a location of an object, and obtain distance data based on the measured result. The distance sensor <NUM> according to an example may be implemented as a light detection and ranging (LIDAR) sensor or a time of flight (TOF) sensor, but is not limited thereto.

The processor <NUM> may identify whether a dynamic object exists in a space within a certain range from the robot <NUM> based on the distance data obtained by the distance sensor <NUM>, and if it is identified that the dynamic object exists, the processor may end the calibration operation for the camera <NUM>-<NUM>.

The user interface <NUM> is a constituent element involved in performing an interaction of the robot <NUM> with the user. For example, the user interface <NUM> may include at least one of a touch sensor, a motion sensor, a button, a jog dial, a switch, a microphone, or a speaker, but is not limited thereto.

The communication interface <NUM> may input and output various types of data. For example, the communication interface <NUM> may transmit and receive various types of data with an external device (e.g., source device), an external storage medium (e.g., a USB memory), and an external server (e.g., web hard) through a communication method such as AP-based wireless-fidelity (Wi-Fi) (wireless LAN network), Bluetooth, Zigbee, wired/wireless local area network (LAN), wide area network (WAN), Ethernet, IEEE <NUM>, high-definition multimedia interface (HDMI), universal serial bus (USB), mobile high-definition link (MHL), Audio Engineering Society/ European Broadcasting Union (AES/EBU), Optical, Coaxial, and the like.

If a user command related to the calibration for the camera <NUM>-<NUM> is received through at least one of the user interface <NUM> or the communication interface <NUM>, the processor <NUM> may control the driving unit <NUM> so that the robot <NUM> moves to the predetermined point for calibration.

<FIG> are diagrams illustrating a predetermined point according to an embodiment.

According to <FIG>, the robot <NUM> may dock at the charge station <NUM> and then perform the calibration for the plurality of sensors <NUM> simultaneously while performing the charging (<NUM>). In this case, the robot <NUM> may orient the plurality of sensors <NUM> in a certain direction in a long-period stationary state. Herein, the robot <NUM> may perform the calibration for the plurality of sensors <NUM> in a state where the plurality of sensors <NUM> are oriented in a specific direction set based on the charge station <NUM>. For example, the specific direction may include at least one direction of a direction facing the charge station <NUM> or the opposite direction thereof, but is not limited thereto.

In this case, the memory <NUM> may store a reference image corresponding to an object located in a direction in which the plurality of sensors <NUM> are oriented while the robot <NUM> docks at the charge station <NUM>. Referring to <FIG>, the memory <NUM> may store the reference image corresponding to one side of the charge station <NUM> and the processor <NUM> may determine whether it is necessary to perform the calibration for the plurality of sensors <NUM> based on the reference image stored in the memory <NUM> and a plurality of images obtained through the plurality of sensors <NUM>.

Specifically, the reference image corresponding to one side of the charge station <NUM> may include a depth image and the processor <NUM> may obtain a depth image based on the plurality of images obtained through the plurality of sensors <NUM> and determine whether it is necessary to perform the calibration for the plurality of sensors <NUM> by comparing the obtained depth image with the reference image.

Referring to <FIG>, the robot <NUM> may perform the calibration for the plurality of sensors <NUM> at a point spaced apart from a wall surface <NUM> with no features by a threshold distance d. The plurality of sensors <NUM> according to an embodiment may include a distance sensor, and if it is identified that the wall surface <NUM> is a wall surface with no complicated pattern based on an image and distance data obtained through the plurality of sensors <NUM>, the processor <NUM> may control the driving unit <NUM> so that the robot <NUM> is spaced apart from the wall surface <NUM> by the threshold distance d. In addition, the processor <NUM> may control the driving unit <NUM> so that the plurality of sensors <NUM> are oriented in the direction of the wall surface <NUM>.

In this case, the memory <NUM> may store a reference image corresponding to the wall surface orthogonal to the ground that is spaced apart from the robot <NUM> by the threshold distance d. The processor <NUM> may perform the calibration for the plurality of sensors <NUM> by comparing the reference image stored in the memory <NUM> with a depth image obtained by synthesizing the plurality of images obtained through the plurality of sensors <NUM>.

Referring to <FIG>, if it is identified that there are no obstacles in a space of a certain range <NUM> from the robot <NUM>, the robot <NUM> may perform the calibration for the plurality of sensors <NUM>. If it is identified that there are no obstacles in the space of the certain range <NUM> from the robot <NUM> based on sensing data obtained through the plurality of sensors <NUM>, the processor <NUM> may control the driving unit <NUM> to stop the robot <NUM> and perform the calibration for the plurality of sensors <NUM> based on the image obtained through the plurality of sensors <NUM>.

In this case, the memory <NUM> may store a reference image corresponding to a floor surface on which the robot <NUM> is located. The processor <NUM> may perform the calibration for the plurality of sensors <NUM> by comparing the reference image stored in the memory <NUM> with the depth image obtained by synthesizing the plurality of images obtained through the plurality of sensors <NUM>.

In <FIG>, it is described that the processor <NUM> performs the calibration for the plurality of sensor <NUM> by comparing the reference image stored in the memory <NUM> with the depth image, but the processor <NUM> according to another example may determine whether it is necessary to perform the calibration for the plurality of sensors <NUM> without comparing the reference image with the depth image.

The processor <NUM> according to another example may identify a flat surface included in any one of the charge station <NUM>, the wall surface <NUM> with no features, or the floor surface within the certain range <NUM> from the robot <NUM> based on the depth image obtained by synthesizing the plurality of images obtained through the plurality of sensors <NUM>. For example, the processor <NUM> may identify the flat surface included in any one of the charge station <NUM>, the wall surface <NUM> with no features, or the floor surface within the certain range <NUM> from the robot <NUM> based on a RANdom SAmple Consensus (RANSAC) algorithm. Specifically, the processor <NUM> may identify a flat surface including the largest number of point clouds within the threshold distance, by repeating an operation of extracting three random point clouds among a plurality of point clouds included in the depth image, identifying a flat surface including the three extracted point clouds, and calculating the number of point clouds located within the threshold distance from the identified flat surface.

If the flat surface included in any one of the charge station <NUM>, the wall surface <NUM> with no features, or the floor surface within the certain range <NUM> from the robot <NUM> is identified based on the depth image, the processor <NUM> may identify whether a threshold value or more of point clouds are included within the threshold range from the identified flat surface. If the threshold value or more of the point clouds are not included within the threshold range from the flat surface, the processor <NUM> may determine that it is necessary to perform the calibration for the plurality of sensors <NUM>.

However, the method for determining whether it is necessary to perform the calibration is merely an example, and the processor <NUM> according to various embodiments of the invention may determine whether it is necessary to perform the calibration for the plurality of sensors <NUM> based on a method different from the method described with reference to <FIG>. <FIG> and <FIG> are diagrams illustrating a sensor calibration operation according to various embodiments. Referring to <FIG> and <FIG>, an operation of the robot <NUM> will be described by assuming that the plurality of sensors <NUM> includes a first camera and a second camera.

Referring to <FIG>, if it is identified that the predetermined event has occurred, the robot <NUM> according to an embodiment of the invention may move to the predetermined point and obtain a plurality of images through the plurality of sensors <NUM>. In addition, the processor <NUM> may obtain a depth image based on a first image <NUM> and a second image <NUM> obtained through the plurality of sensors <NUM> (S411). For example, the processor <NUM> may obtain the depth image based on the first image <NUM>, the second image <NUM>, and synthesis data stored in the memory <NUM>. Herein, the synthesis data may include a plurality of matrix data (camera matrix) including intrinsic parameters related to optical characteristics and scaling of the first camera and the second camera and extrinsic parameters related to the mechanical distortion of at least one camera of the first camera or the second camera, but is not limited thereto.

In addition, the processor <NUM> may compare the obtained depth image with a reference image corresponding to the predetermined point stored in the memory <NUM> (S412).

Then, the processor <NUM> may identify whether a similarity between the depth image and the reference image is equal to or more than the threshold value (S413). If the similarity between the depth image and the reference image is identified to be equal to or more than the threshold value (S413: Y), the processor <NUM> may determine that it is not necessary to perform the calibration for the plurality of sensors <NUM> and end the sensor calibration process (S414).

On the other hand, if the similarity between the depth image and the reference image is identified to be less than the threshold value (S413: N), the processor <NUM> may obtain calibration data for calibrating any one camera of the first camera or the second camera and store this in the memory <NUM> (S415). For example, the processor <NUM> may identify that there is the mechanical distortion on the second camera based on the first camera, and obtain the calibration data for calibrating the second camera. Herein, the calibration data may include a target value for adjusting the extrinsic parameters included in the synthesis data, but is not limited thereto.

For example, the processor <NUM> may adjust at least one extrinsic parameter value related to at least one of the x axis direction shift <NUM>, the y axis direction shift <NUM>, the z axis direction shift <NUM>, the x axis center rotation <NUM>, the y axis center rotation <NUM>, and the z axis center rotation <NUM> described above with reference to <FIG>, obtain a depth image based on synthesis data with the adjusted extrinsic parameter value, the first image <NUM>, and the second image <NUM>, and identify an extrinsic parameter value derived with a highest similarity based on an algorithm including a process of identifying a similarity between the obtained depth image and the reference image stored in the memory <NUM>. In addition, the processor <NUM> may obtain calibration data in which the identified extrinsic parameter value is used as a target value.

After obtaining the calibration data, the processor <NUM> may additionally obtain sensing data including images obtained through the first camera and the second camera. In addition, the processor <NUM> may obtain sensing data calibrated by applying the calibration data to the additionally obtained sensing data. For example, the processor <NUM> may adjust the extrinsic parameter included in the synthesis data based on the calibration data, and obtain sensing data calibrated based on the sensing data obtained through the second camera that is identified as requiring the calibration and the synthesis data with the adjusted extrinsic parameter.

If it is identified that the calibrated sensing data is improved than the sensing data before the calibration by the threshold value or more, the processor <NUM> may store the obtained calibration data in the memory <NUM>. For example, the processor <NUM> may obtain a new depth image based on the sensing data obtained through the first camera and the calibrated sensing data corresponding to the second camera, and if a similarity between the new depth image and the reference image stored in the memory <NUM> is increased from the similarity between the existing depth image and the reference image by the threshold value or more, the processor may store the obtained calibration data in the memory <NUM>.

Meanwhile, when the predetermined event has occurred again, the processor <NUM> may obtain the depth image based on the synthesis data in which the extrinsic parameter is adjusted based on the calibration data stored in the memory <NUM> through the above process, the first image <NUM>, and the second image <NUM>, and determine whether it is necessary to perform the calibration for the plurality of sensors <NUM> by identifying whether the quality of the obtained depth image is excellent. If it is identified again that it is necessary to perform the calibration for the plurality of sensors <NUM>, the processor <NUM> may obtain new calibration data and update the calibration data obtained previously.

Referring to <FIG>, if it is identified that the predetermined event has occurred, the robot <NUM> according to another embodiment of the invention may move to the predetermined point and obtain a plurality of images through the plurality of sensors <NUM>. In addition, the processor <NUM> may obtain the depth image based on the first image <NUM> and the second image <NUM> obtained through the plurality of sensors <NUM> (S421).

In addition, the processor <NUM> may identify a flat surface included in the predetermined point based on the obtained depth image (S422). For example, the processor <NUM> may identify the flat surface included in any one of the charge station <NUM>, the wall surface with no features, or the floor surface within the certain range from the robot <NUM> by applying the RANSAC algorithm to the plurality of point clouds included in the depth image.

Then, the processor <NUM> may identify whether the threshold value or more of point clouds are included within the threshold range from the identified flat surface (S423). Herein, the threshold value may be a predetermined value in relation to the number of point clouds included in the threshold range from the identified flat surface or a rate of the point clouds within the threshold range from the identified flat surface among all point clouds, but is not limited thereto.

If it is identified that the threshold value or more of point clouds are included in the threshold range from the identified flat surface (S423: Y), the processor <NUM> may identify that it is not necessary to perform the calibration for the plurality of sensors <NUM> and end the sensor calibration process (S424).

On the other hand, if it is identified that the threshold value or more of point clouds are not included within the threshold range from the identified flat surface (S423: N), the processor <NUM> may obtain calibration data for calibrating any one of the first camera and the second camera and store this in the memory <NUM> (S425). <FIG> is a diagram illustrating calibration performed based on a user command of the robot according to an embodiment.

Referring to <FIG>, the robot <NUM> may include the user interface <NUM> and the communication interface <NUM>.

The robot <NUM> may receive various types of user commands related to the calibration of the plurality of sensors <NUM> through the user interface <NUM>. For example, the processor <NUM> may receive a user command for instructing the calibration for the plurality of sensors <NUM> through the user interface <NUM> and control the driving unit <NUM> so that the robot <NUM> moves to the charge station <NUM> based on the received user command.

In addition, when the processor <NUM> receives the user command for instructing the end of calibration for the plurality of sensors <NUM> through the user interface <NUM>, the processor <NUM> may end the calibration for the plurality of sensors <NUM> being performed based on the user command.

The user may input the user command related to the calibration of the plurality of sensors <NUM> included in the robot <NUM> through a user terminal <NUM>. In this case, the user terminal <NUM> may display a guide graphical user interface (GUI) <NUM> related to the calibration for the plurality of sensors <NUM> and transmit a control signal corresponding to the user command to the robot <NUM>.

The communication interface <NUM> may receive the control signal corresponding to the user command from the user terminal <NUM>. When the control signal corresponding to the user command is received through the communication interface <NUM>, the processor <NUM> may control the driving unit <NUM> so that the robot <NUM> moves to the charge station <NUM>.

Accordingly, the robot <NUM> may perform the calibration for the plurality of sensors <NUM> based on the user command received through the user interface <NUM> or the communication interface <NUM>, and accordingly, although the predetermined time interval or the interval in which a predetermined number of tasks has been performed has not arrived, the calibration for the plurality of sensors <NUM> may be performed.

<FIG> and <FIG> are sequence diagrams illustrating a system according to various embodiments.

Referring to <FIG>, a system <NUM> including the robot <NUM> including the plurality of sensors <NUM> and the charge station <NUM> may perform calibration for the plurality of sensors <NUM> according to organic operations between the constituent elements <NUM> and <NUM>.

The charge station <NUM> may supply power to the robot <NUM>. Specifically, if it is identified that the robot <NUM> has docked at the charge station <NUM>, the charge station <NUM> may supply power to the robot <NUM>. For this, the charge station <NUM> may include a power supplier, a combination portion for docking of the robot <NUM>, and a controller which controls the power supplier to supply the power to the robot <NUM>, if it is identified that the robot <NUM> has docked at the combination portion, but is not limited thereto.

If it is identified that at least one of the predetermined time interval or the interval in which a predetermined number of tasks has been performed has arrived, the robot <NUM> according to an example may determine that the predetermined event has occurred (S611).

In addition, the robot <NUM> may move to the charge station <NUM> (S612) and dock at the charge station <NUM>. In addition, if it is identified that the robot <NUM> has moved to the charge station <NUM>, the charge station <NUM> may transmit a signal for instructing whether to dock to the robot <NUM> (S613) and supply the power to the robot <NUM>.

When signal for instructing the docking is received from the charge station <NUM>, the robot <NUM> may identify whether it is necessary to perform the calibration for at least one sensor of the plurality of sensors <NUM>, and if it is identified that it is necessary to perform the calibration for the at least one sensor, the robot <NUM> may perform the calibration for the plurality of sensors <NUM> based on a plurality of images obtained from the plurality of sensors <NUM> (S614).

Referring to <FIG>, a system <NUM>' including the robot <NUM> including the plurality of sensors <NUM>, the charge station <NUM>, and the user terminal <NUM> may perform the calibration for the plurality of sensors <NUM> according to organic operations between the constituent elements <NUM>, <NUM>, and <NUM>.

The user terminal <NUM> may be implemented as an electronic device. For example, the user terminal <NUM> may be implemented in various forms of a smartphone, a tablet PC, a mobile phone, a video phone, an e-book reader, a desktop personal computer (PC), a laptop personal computer (PC), a netbook computer, a workstation, a server, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a medical device, a camera, a wearable device, or the like, but is not limited thereto.

A user terminal <NUM> according to an example may receive a user command related to the calibration for the plurality of sensors <NUM> included in the robot <NUM> (S621). The user terminal <NUM> may transmit a control signal corresponding to the user command to the robot <NUM> (S622).

The robot <NUM> may move to the charge station <NUM> (S623) and dock at the charge station <NUM> based on the control signal received from the user terminal <NUM>. In addition, if it is identified that the robot <NUM> has moved to the charge station, the charge station <NUM> may transmit a signal for instructing the docking to the robot <NUM> (S624) and supply the power to the robot <NUM>.

If the signal for instructing the docking is received from the charge station <NUM>, the robot <NUM> may identify whether it is necessary to perform the calibration for at least one sensor among the plurality of sensors <NUM>, and if it is identified that it is necessary to perform the calibration for the at least one sensor, the robot may perform the calibration for the plurality of sensors <NUM> based on the plurality of images obtained from the plurality of sensors <NUM> (S625).

In addition, when the calibration for the plurality of sensors <NUM> is completed, the robot <NUM> may transmit information on a sensor calibration result to the user terminal <NUM> (S626). Although not illustrated in <FIG>, the user terminal <NUM> may provide a GUI for the sensor calibration result to the user based on the information received from the robot <NUM>.

<FIG> is a flowchart illustrating a control method according to an embodiment.

Referring to <FIG>, in the control method according to an embodiment of the invention, if it is identified that the predetermined event has occurred, the robot is moved to the predetermined point (S710).

If it is identified that the robot has moved to the predetermined point, a plurality of images may be obtained through the plurality of sensors (S720).

It is identified whether it is necessary to perform the calibration for at least one sensor of the plurality of sensors based on the plurality of obtained images (S730).

If it is identified that it is necessary to perform the calibration for the at least one sensor, the calibration data for calibrating the sensing data corresponding to the at least one sensor may be obtained based on the plurality of images and stored (S740).

When the sensing data is obtained from the at least one sensor, the obtained sensing data may be calibrated based on the stored calibration data (S750).

Finally, the robot may be driven based on the calibrated sensing data (S760).

The identifying whether it is necessary to perform the calibration (S730) may include identifying whether there is a mechanical distortion on each of the plurality of sensors based on the plurality of obtained images, and based on the identifying that there is the mechanical distortion on at least one sensor of the plurality of sensors, identifying that it is necessary to perform the calibration for the at least one sensor.

In addition, the identifying whether it is necessary to perform the calibration (S730) may include obtaining a depth image based on the plurality of obtained images, comparing a reference image related to the predetermined point with the obtained depth image, and identifying whether it is necessary to perform the calibration for the at least one sensor of the plurality of sensors based on the comparison result.

In addition, in the moving the robot (S710), if it is identified that at least one of the predetermined time interval or the interval in which a predetermined number of tasks has been performed has arrived, it is identified that the predetermine event has occurred.

In the obtaining the plurality of images (S720), if it is identified that the robot has docked at the charge station, it may be identified that the robot has moved to the predetermined point.

The control method may further include identifying whether a dynamic object exists in a surrounding environment of the predetermined point, after identifying that the robot has moved to the predetermined point, and based on the dynamic object being identified to exist, finishing the calibration operation for the plurality of sensors.

In the moving the robot (S710), when a user command is received, the robot may move to the predetermined point.

In the calibrating the obtained sensing data (S750) may include, based on the calibration data being obtained, additionally obtaining sensing data from the at least one sensor, obtaining calibrated sensing data by applying the calibration data to the additionally obtained sensing data, and, based on the identifying that the calibrated sensing data is improved than the sensing data by a threshold value, storing the obtained calibration data in the memory.

The control method may further include, based on an event in which a traveling mode of the robot is changed occurring after identifying that the robot has moved to the predetermined point, finishing the calibration operation for the plurality of sensors.

<FIG> and <FIG> are diagrams illustrating an event that is able to occur during the sensor calibration according to an embodiment.

<FIG> is a diagram illustrating the operation of the robot for a dynamic object identification event according to an embodiment.

Referring to <FIG>, the robot <NUM> according to an example may further include a distance sensor. If it is identified that the predetermined event has occurred and it is identified that the robot <NUM> has moved to the predetermined point, the processor <NUM> may proceed with the calibration for the plurality of sensors <NUM> (S811).

In addition, the processor <NUM> may identify whether a dynamic object exists in the surrounding environment of the predetermined point based on distance data obtained by the distance sensor in the process of proceeding the calibration for the plurality of sensors <NUM> (S812). When the dynamic object exists (S812: Y), an accuracy of the sensor calibration may be deteriorated according to a change of a location of the robot <NUM> and a direction in which the plurality of sensors <NUM> are oriented due to the dynamic object, the processor <NUM> may finish the calibration while not completing the sensor calibration (S814).

When the dynamic object does not exist (S812: N), the processor <NUM> may perform the calibration by identifying whether it is necessary to perform the calibration for the plurality of sensors <NUM>. The processor <NUM> may identify whether the calibration for the plurality of sensors <NUM> is completed (S813), and if it is identified that the calibration is completed (S813: Y), the processor may store the calibration data corresponding to at least one sensor of the plurality of sensors <NUM> in the memory <NUM> and finish the calibration (S814).

On the other hand, if it is identified that the calibration is not completed (S813: N), the processor <NUM> may identify whether the predetermined event has occurred, and if it is identified that the predetermined event has occurred, the processor may proceed the sensor calibration again (S811).

<FIG> is a diagram illustrating the operation of the robot for a traveling mode change identification event according to an embodiment.

Referring to <FIG>, in an example, the processor <NUM> may identify that the predetermined event has occurred, and if it is identified that the robot <NUM> has moved to the predetermined point, the processor may proceed the calibration for the plurality of sensors <NUM> (S821).

In addition, the processor <NUM> may identify whether the traveling mode of the robot <NUM> is changed in the process of proceeding the calibration for the plurality of sensors <NUM> (S822). If the traveling mode of the robot <NUM> is changed (S822: Y), the robot <NUM> needs to provide the service corresponding to the changed traveling mode, and accordingly, the processor <NUM> may finish the sensor calibration while not completing the sensor calibration (S824).

On the other hand, if the traveling mode of the robot <NUM> is changed (S822: N), the processor <NUM> may perform the calibration by identifying whether it is necessary to perform the calibration for the plurality of sensors <NUM>. The processor <NUM> may identify whether the calibration for the plurality of sensors <NUM> is completed (S823), if it is identified that the calibration is completed (S823: Y), the processor may store the calibration data corresponding to the at least one sensor of the plurality of sensors <NUM> in the memory <NUM> and finish the calibration (S824).

On the other hand, if it is identified that the calibration is not completed (S823: N), the processor <NUM> may identify whether the predetermined event has occurred, and if it is identified that the predetermined event has occurred, the processor may proceed the sensor calibration again (S821).

<FIG> is a diagram illustrating a calibration data storage operation of the robot according to an embodiment.

Referring to <FIG>, if it is identified that the predetermined event has occurred and it is identified that the robot <NUM> has moved to the predetermined point and it is necessary to perform the calibration for at least one sensor of the plurality of sensors <NUM>, the processor <NUM> may perform the calibration for the plurality of sensors <NUM> (S910).

For example, the processor <NUM> may obtain calibration data corresponding to at least one sensor of the plurality of sensors <NUM> based on a plurality of images obtained by the plurality of sensors <NUM>, and apply the calibration data to the sensing data additionally obtained by the plurality of sensors <NUM>.

In addition, the processor <NUM> may identify whether the calibrated sensing data obtained by applying the calibration data to the additionally obtained sensing data is improved than the existing sensing data (S920). For example, if a similarity between a depth image obtained based on the calibrated sensing data and a reference image stored in the memory <NUM> is increased than before the calibration by a threshold value or more, the processor <NUM> may determine that the calibrated sensing data is improved than the existing sensing data.

If it is identified that the calibrated sensing data is improved than the existing sensing data by the threshold value or more (S920: Y), the processor <NUM> may store the obtained calibration data in the memory <NUM> (S930) and finish the calibration for the plurality of sensors <NUM> (S940).

On the other hand, if it is not identified that the calibrated sensing data is improved than the existing sensing data by the threshold value or more (S920: N), the processor <NUM> may not store the obtained calibration data in the memory <NUM> and finish the calibration for the plurality of sensors <NUM> (S940). For example, if hardware defect related to the mechanical distortion of the at least one sensor of the plurality of sensors <NUM>, while not related to extrinsic parameters, has occurred (e.g., damage of sensor), the processor <NUM> may finish the calibration for the plurality of sensors <NUM> without storing the obtained calibration data. In this case, the processor <NUM> may obtain error information for the at least one sensor of the plurality of sensors <NUM> and provide the obtained error information to the user.

Through the series of operations described above, if the calibration for the plurality of sensors <NUM> is not accurately performed or there is no significant utility obtained through the sensor calibration, the robot <NUM> may not finish the calibration for the plurality of sensors <NUM>, thereby preventing unnecessary calibration.

Meanwhile, the methods according to the embodiments of the invention described above may be implemented in a form of an application installable in the robot of the related art.

In addition, the methods according to the embodiments of the invention described above may be implemented simply by the software upgrade or hardware upgrade in the robot of the related art.

Further, the embodiments of the invention described above may be performed through an embedded server provided in the robot or at least one external server.

The embodiments described above may be implemented in a recording medium readable by a computer or a similar device using software, hardware, or a combination thereof. In some cases, the embodiments described in this specification may be implemented as the processor <NUM> itself. According to the implementation in terms of software, the embodiments such as procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described in this specification.

Computer instructions for executing processing operations of the robot <NUM> according to the embodiments of the invention descried above may be stored in a non-transitory computer-readable medium. When the computer instructions stored in such a non-transitory computer-readable medium are executed by the processor of a specific machine, the computer instructions of the robot <NUM> according to various embodiments described above may be executed by the specific machine.

The non-transitory computer-readable medium is not a medium storing data for a short period of time such as a register, a cache, or a memory, but may refer to a medium that semi-permanently stores data and is readable by a machine. Specific examples of the non-transitory computer-readable medium may include a CD, a DVD, a hard disk drive, a Blu-ray disc, a USB, a memory card, and a ROM.

Claim 1:
A robot (<NUM>, <NUM>') comprising:
a plurality of sensors (<NUM>);
a memory (<NUM>);
a driving unit (<NUM>); and
a processor (<NUM>) configured to:
based on identifying that a predetermined event has occurred, control the driving unit (<NUM>) so that the robot (<NUM>, <NUM>') moves to a predetermined point corresponding to a reference image,
based on identifying that the robot (<NUM>, <NUM>') has moved to the predetermined point, obtain a plurality of images through the plurality of sensors (<NUM>),
identify whether it is necessary to perform calibration for at least one sensor of the plurality of sensors (<NUM>) based on the reference image and the obtained plurality of images,
based on identifying that it is necessary to perform the calibration for the at least one sensor,
obtain calibration data to calibrate sensing data corresponding to the at least one sensor based on a synthesis of the plurality of images and store the obtained calibration data in the memory (<NUM>),
based on the sensing data being obtained from the at least one sensor, calibrate the obtained sensing data based on the calibration data stored in the memory (<NUM>), and
control the driving unit (<NUM>) based on the calibrated sensing data,
wherein the memory (<NUM>) is configured to store the reference image related to the predetermined point, and
wherein the processor (<NUM>) is further configured to:
obtain a depth image based on a synthesis of the plurality of obtained images,
compare the reference image with the obtained depth image, and
identify whether it is necessary to perform the calibration for the at least one sensor (<NUM>) of the plurality of sensors (<NUM>) based on a result of the comparison.