Patent ID: 12229875

DESCRIPTION OF EMBODIMENTS

A display data generation apparatus10according to one or more embodiments of the present disclosure is described in detail below with reference to the drawings.

Embodiment 1

As illustrated inFIG.1, the display data generation apparatus10according to the present embodiment is included in a display system100together with an imager20and a device30. The imager20captures images of the device30, and the device30is controlled by a controller31. The display system100is used for abnormality analysis at a facility such as a factory. An abnormality herein refers to a state that is determined, by a manager, to fall out of a normal operating range of the facility. For example, damage to a workpiece processed by the device30and a deviation of the movement path of the workpiece from a pre-designed flow line correspond to such abnormalities. One abnormality may cause another abnormality. For example, an abnormality in a workpiece may cause an abnormal state of the device30.

As illustrated inFIG.2, the display system100combines a moving image101of the environment of the device30and a moving image102of a three-dimensional (3D) model drawn based on the operation log of the device30and displays the resultant image on a screen103. This assists abnormality analysis performed by an analyst.

The imager20is a camera with an image sensor. The imager20is, for example, a camera recorder connected to a programmable logic controller (PLC), or a surveillance camera that operates independently of the PLC. The imager20periodically transmits imaging data indicating moving images including frame images of the device30and the surroundings of the device30to the display data generation apparatus10through an industrial network. The frame rate of the moving images captured by the imager20is, for example, 10 frames per second (fps) or 30 fps. The imager20may capture moving images with visible light or moving images with infrared light.

The device30is a factory automation (FA) device including movable parts. The device30described in the example below is mainly an industrial robot including an arm with four axes of freedom of motion. The device30operates in accordance with a control command from the controller31. More specifically, as illustrated in the upper part ofFIG.2, the device30repeatedly performs a series of operations including gripping a workpiece300placed on a tray301, moving the workpiece300into an inspector302, gripping the workpiece300again after inspection, and moving the workpiece300to a tray303.

As illustrated inFIG.3, the workpiece300is to be placed on the tray301with the axis maintained horizontally. In other words, as illustrated inFIG.4that is a cross-sectional view taken along line AA′ inFIG.3, the workpiece300having one end on the left and the other end on the right being horizontal is placed on the tray301properly.

As illustrated inFIG.5, the workpiece300having one end and the other end tilted with respect to the tray301is placed on the tray301improperly. Such placement may cause an abnormality in the gripping state of the device30, an inappropriate inspection in the inspector302, and an abnormal stop of the device30. However, a single captured image as illustrated in the upper part ofFIG.2does not allow easy determination of the placement state as the cause of the abnormalities. Although the operation record of the device30is commonly taken, the orientation of the workpiece300is not usually recorded as a sensing target.

When a sensor304for determining the presence or absence of the workpiece300at the position indicated by the white circle inFIGS.4and5, the output from the sensor304is on for the workpiece300placed properly, and the output from the sensor304is off for the workpiece300placed improperly. Referring to the above output allows easier determination of an abnormality associated with the tray301than when the sensor304is eliminated.

However, determining whether no workpiece300is on the tray301or the workpiece300has an abnormal orientation involves examining a captured image. When the image indicates that the workpiece300has an abnormal orientation, such a single captured image may not easily allow determination as to whether the abnormal orientation has been caused by another abnormality and whether the abnormal orientation is the cause of another abnormality.

Referring back toFIG.1, the controller31may be a PLC. The controller31executes a predetermined ladder program to control the device30. More specifically, the controller31controls, with a servo amplifier, the values of parameters such as rotation angles and angular velocities of each of the four axes included in the device30. The controller31includes a recorder unit that records a log being a record of the values of the parameters. The controller31transmits log data indicating the log to the display data generation apparatus10through a communication line such as a universal serial bus (USB) cable.

The display data generation apparatus10is a computer such as an industrial personal computer (PC) and a tablet terminal. As illustrated inFIG.6, the display data generation apparatus10includes, as hardware components, a processor41, a main storage42, an auxiliary storage43, an input device44, an output device45, and a communicator46. The main storage42, the auxiliary storage43, the input device44, the output device45, and the communicator46are connected to the processor41with an internal bus47.

The processor41includes a central processing unit (CPU) or a micro processing unit (MPU) as a processing circuit. The processor41executes a program48stored in the auxiliary storage43to implement various functions of the display data generation apparatus10and perform processing described later. The program48corresponds to an example of a display data generation program.

The main storage42includes a random-access memory (RAM). The program48is loaded from the auxiliary storage43into the main storage42. The main storage42is used as a work area for the processor41.

The auxiliary storage43includes a nonvolatile memory, such as an electrically erasable programmable read-only memory (EEPROM) and a hard disk drive (HDD). The auxiliary storage43stores, in addition to the program48, various data items used for the processing performed by the processor41. The auxiliary storage43provides data usable by the processor41to the processor41as instructed by the processor41, and stores data provided by the processor41.

The input device44includes, for example, a keyboard or a pointing device. The input device44acquires information input by the user, who is the analyst, of the display data generation apparatus10and provides the acquired information to the processor41.

The output device45includes, for example, a light-emitting diode (LED), a liquid crystal display (LCD), or a speaker. The output device45presents various items of information to the user as instructed by the processor41.

The communicator46includes a communication interface circuit for transmitting and receiving signals to and from an external device. The communicator46receives a signal from the external device and outputs data indicated by the signal to the processor41. The communicator46also transmits a signal indicating data output from the processor41to the external device.

With the hardware described above operating in cooperation, the display data generation apparatus10performs various functions. More specifically, as illustrated inFIG.1, the display data generation apparatus10includes, as functional components, a moving image acquirer11that acquires a moving image of the device30captured by the imager20, a model acquirer12that acquires model data indicating a 3D model of the device30, a log acquirer13that acquires the log data indicating a log of the operation of the device30, an estimator14that estimates the position of the imager20from the captured moving image, a display controller15that generates display data indicating details to be displayed and causes a display16to display the details, and the display16including a screen103on which the details in the display data appear. The display16is mainly implemented by the output device45.

The moving image acquirer11is mainly implemented by the communicator46. The moving image acquirer11requests the imager20to provide a moving image captured in a time range including a time specified by the user to receive imaging data indicating the moving image from the imager20. The moving image captured may hereafter be referred to as a captured moving image. The moving image acquirer11may acquire the imaging data by reading the imaging data from a non-transitory recording medium such as a memory card or from an external server device, rather than by receiving the imaging data from the imager20. The captured moving image corresponds to an example of a first moving image of an environment including a device. The moving image acquirer11corresponds to an example of moving image acquisition means for acquiring imaging data indicating the first moving image in the display data generation apparatus10.

FIG.7schematically illustrates example imaging data. As illustrated inFIG.7, the imaging data indicates multiple frame images in a manner associated with the respective capturing dates and times. The imaging data may include a value indicating the start date and time when the first frame image is captured and a value indicating a frame rate to substantially indicate the capturing date and time of each frame image. When the imaging data includes these values, the date and time when each frame image is captured is calculated from the start date and time, the frame rate, and the frame number. The method for indicating the capturing time of each frame image is not limited to the above, and may be changed as appropriate.

Referring back toFIG.1, the model acquirer12is mainly implemented by the processor41, the input device44, and the communicator46operating in cooperation with one another. The model acquirer12acquires, from the user, the model data indicating the 3D model simulating the shape of the device30. More specifically, the model acquirer12acquires the model data by reading the model data from an address in a non-transitory recording medium or the external server device specified by the user. The device30has the shape predefined by a mechanical design using a 3D computer-aided design (CAD) software application. The model data is generated using such a mechanical design, and indicates the spatial coordinates of points, lines, and surfaces defining the 3D model.

FIG.8schematically illustrates example model data. As illustrated inFIG.8, the model data indicates the 3D model of the device using the spatial coordinates and also indicates object positions. Although a 3D model of a planar support base supporting the device30is also illustrated inFIG.8, the 3D model of the support base may be eliminated. An object position indicates a position corresponding to an object in the captured moving image. In the example inFIG.8, the positions of three marks being objects are indicated as object positions. The object positions are used to determine the scale and the angle of the 3D model. The determination of the scale and the angle based on the object positions is described in detail later. The model acquirer12corresponds to an example of model acquisition means for acquiring model data indicating a 3D model of a device and an object position corresponding to an object in the display data generation apparatus10.

The log acquirer13is mainly implemented by the communicator46. The log acquirer13requests the controller31to provide a log recorded in the time range including the time specified by the user to receive log data from the controller31. The time specified by the user is equal to the time specified for the moving image acquirer11. Thus, the time range in which the log acquired by the log acquirer13is generated overlaps the time range in which the captured moving image acquired by the moving image acquirer11is captured. The log acquirer13may acquire the log data differently, rather than by receiving the log data from the controller31. The log acquirer13may acquire the log data by reading the log data from a non-transitory recording medium or the external server device, or may acquire the log data from the device30when the device30records a log. The log acquirer13corresponds to an example of log acquisition means for acquiring log data indicating a log of an operation of the device when the first moving image is captured in the display data generation apparatus10.

The log data is information to reproduce the motion of the movable parts in the device30.FIG.9illustrates example log data. In the example inFIG.9, the log data indicates, with one rotation being 360 degrees, the angles of the first, second, third, and fourth axes of the device30in chronological order in a manner associated with the date and time of recording. Each record included in the log data may be a regular record or an irregular record.

The estimator14is mainly implemented by the processor41. The estimator14acquires the captured moving image from the moving image acquirer11and estimates the position and the orientation of the imager20based on the objects in the captured moving image. More specifically, as illustrated inFIG.10, the estimator14detects, from the captured moving image, the three mutually different marks pre-attached to the support base for the device30as objects. The estimator14then estimates the position and the orientation of the imager20based on the coordinates on the image including the detected objects. The estimation is performed using a known solution algorithm for the perspective-n-point (PnP) problem. The coordinates of the marks attached as objects are known. The position and the orientation of the imager20are defined in the same coordinate system as the coordinates of the mark. The estimator14corresponds to an example of estimation means for estimating the position and the orientation of an imager that captures the first moving image based on the object in the first moving image in the display data generation apparatus10.

The display controller15is mainly implemented by the processor41. The display controller15acquires the model data from the model acquirer12and the log data from the log acquirer13and changes the 3D model based on the log. More specifically, the display controller15changes, in the 3D model, the movable part models corresponding to the movable parts based on the values indicated by the log. For example, the display controller15rotates the movable part model corresponding to the first axis based on the record illustrated inFIG.9to match the angle of the movable part model with 30 degrees indicated by the log. The display controller15also rotates the movable part models for the other axes in the same manner. The display controller15rotates the movable part models by sequentially using the records with the other recording dates and times in the 3D model.

The display controller15virtually places the 3D model that changes based on the log as described above with respect to the object positions. The objects are attached to the support base stationary in the space to define a coordinate system in the real environment. The object positions in the model data are defined in the coordinate system common to the 3D model. The 3D model is thus defined in the coordinate system substantially common to the device30in the real environment. The display controller15virtually operates the above 3D model based on the log.

The display controller15then acquires, from the estimator14, the result of estimation performed by the estimator14and projects the 3D model at the estimated position and in the estimated orientation to generate a moving image. In other words, the display controller15generates a moving image of the changing 3D model from the same position and orientation as the imager20that has captured the images of the device30. The moving image acquired by projecting the 3D model may hereafter be referred to as a model moving image. In the example inFIG.10, the coordinates (X1, Y1, Z1) indicate the estimated position of the imager20, and the orientation (θ, ψ) is the estimated orientation, where θ is to the azimuth angle, and ψ is the elevation angle.

Although the projection method for the 3D model may also be used commonly as the projection method from the device30to the captured moving image, these projection methods may be different. For example, the estimator14may estimate the angle of view as an imaging range and the aberration in addition to the position and the orientation described above, and the display controller15may use the estimated angle of view and aberration in the projection of the 3D model. The display controller15may generate the model moving image by parallel projection of the 3D model onto a plane defined by the estimated position and orientation of the imager20. Although the model moving image by the parallel projection has parameters, such as aberrations, different from the captured moving image, the model moving image may be any moving image that allows the user to verify the operation of the device30based on the log.

The display controller15causes a 3D simulator to display the generated model moving image on the display16. The 3D simulator is a software application for three-dimensionally displaying a production facility or a controller and a simulation result of the operation of the production facility or the controller. The display controller15causes the captured moving image being semitransparent to appear synchronously with the model moving image in a manner superimposed on the model moving image in the area on the screen103in the display16in which the model moving image appear. In other words, the display controller15causes frame images in the model moving image generated from a record with a specific recording date and time included in the log to sequentially appear in a manner superimposed on the frame images in the captured moving image captured at a date and time equal to the specific recording date and time. The display controller15generates the display data for displaying the model moving image and the captured moving image to be played synchronously, and outputs the display data to the display16, thus causing these moving images to appear on the screen in the display16in a superimposed manner.

For the frame images in the model moving image and the frame images in the captured moving image displayed simultaneously, the recording date and time of the log for generating the model moving image and the capturing date and time of the captured moving image may not completely match, and may have a permissible synchronization error. For example, the synchronization error may be 10 or 100 ms. The synchronization error is permitted unless the error affects the abnormality analysis when the analyst simultaneously views and compares the model moving image and the captured moving image. The display controller15may cause the model moving image and the captured moving image to be played simultaneously with a synchronization error within a predetermined range.

The model moving image corresponds to an example of a second moving image acquired by projecting, at a position and in an orientation estimated by the estimation means, a 3D model placed with respect to an object position while changing the 3D model based on the log. The display controller15corresponds to an example of display data generation means for generating and outputting display data for displaying the first moving image and the second moving image to be played synchronously in the display data generation apparatus10.

A display process performed by the display data generation apparatus10is described with reference toFIGS.11and12. The display process illustrated inFIG.11starts when the program48is executed. The display process corresponds to an example of a display data generation method implementable by the display data generation apparatus10.

In the display process, the processor41in the display data generation apparatus10determines whether a time is specified by the user (step S1). The time corresponds to the date and time at which the user intends to play a moving image. The determination in step S1 may be performed as to whether a start time and an end time are specified, whether a time length for playing the moving image is specified along with a start time, or whether a time range before and after a specified time is specified along with the specified time. A time may be specified by the user with a predetermined length of time before and after the specified time. The determination result in step S1 may be affirmative when the time at which at least a part of the moving image is to be played is specified.

When the determination result in step S1 is negative (No in step S1), the display data generation apparatus10repeats the determination in step S1 and waits for the user to specify a time. When the determination result in step S1 is affirmative (Yes in step S1), the moving image acquirer11acquires a captured moving image captured in the range including the time specified in step S1 (step S2). The log acquirer13acquires a log recorded in a range including the time specified in step S1 (step S3). The model acquirer12acquires model data provided by the user (step S4).

The estimator14estimates the position and the orientation of the imager20based on objects in the captured moving image indicated by imaging data acquired in step S2 (step S5). The estimator14may estimate the position and the orientation of the imager20being stationary from the start to the end of the captured moving image based on objects in a single frame image. The estimator14may estimate the position and the orientation of the imager20being stationary based on objects in multiple frame images, thus increasing the estimation accuracy. The estimator14may sequentially estimate the position and the orientation of the imager20not being stationary for each frame image.

The display controller15generates a model moving image by projecting a 3D model indicated by the model data acquired in step S4 at the position and in the orientation estimated in step S5 while changing the 3D model based on the log indicated by log data acquired in step S3 (Step S6). More specifically, the display controller15uses a record corresponding to each of the recording dates and times included in the log in the 3D model, and then projects the 3D model based on the estimation result in step S5 to generate a frame image corresponding to each of the recording dates and times. When the imager20is not stationary and no estimation result is available at the capturing date and time equal to a recording date and time, the display controller15may use the closest estimation result or may use linearly interpolated estimation results before and after the recording data and time.

The display controller15generates display data for synchronously playing the captured moving image and the model moving image, and outputs the generated display data to the display16(step S7). More specifically, the display controller15controls the superimposed captured moving image and model moving image to be played in accordance with user operations on a play button104and a pause button105as illustrated inFIG.12. This causes the display16to display the two moving images based on the display data (step S8). The display process ends.

The processing in each step and the order of the steps in the display process described above may be changed as appropriate. For example, steps S2, S3, and S4 may be performed in a different order or may be performed in parallel. Although step S2 is to be performed before step S5, steps S3 and S4 may be simply performed before step S6.

As described above, the display controller15generates and outputs the display data for displaying the captured moving image and the model moving image to be played synchronously. The model moving image is acquired by projecting, at the position and in the orientation estimated by the estimator14, the 3D model of the device placed with respect to the object position while changing the 3D model based on the log. Thus, the model moving image in which the viewpoint for capturing the 3D model is substantially equal to the viewpoint for capturing the device is synchronously played with the captured moving image. This allows an analyst to easily compare changes in the state of the real environment including the device with a series of operations of the device indicated by the log, and thus to easily determine the relationship between the substance captured in the image other than the device and an abnormality or determine the effect of the substance on the operations of the device. This can thus further assist the analyst who analyzes abnormalities in detail at the FA sites.

The analyst can also examine the state when a defect occurs in a production facility and an FA device with an actual video together with the 3D model. This facilitates identification of the cause of the defect. This allows early recovery from the defect and improves the operation capacity of the FA device.

Embodiment 2

Embodiment 2 is described focusing on the differences from Embodiment 1 described above. The same reference signs denote the components that are the same as or similar to those in Embodiment 1, and such components are not described or are described briefly. The present embodiment differs from Embodiment 1 in emphasizing the result of a comparison between the model moving image and the captured moving image.

As illustrated inFIG.13, the display controller15in the present embodiment extracts an area A11including a set of pixels changeable between frames in a captured moving image and an area A12including a set of pixels unchangeable. An unchangeable pixel may be a pixel having a pixel value with a change smaller than a predetermined threshold. Similarly, the display controller15extracts an area A21including a set of pixels changeable between frames in a model moving image and an area A22including a set of pixels unchangeable. The display controller15then extracts an area A30corresponding to a difference between the changeable area A11in the captured moving image and the changeable area A21in the model moving image. More specifically, a set of pixels in the area A21that are not included in the area A11are extracted as the area A30.

The unchangeable area A12does not change in and around the device30being captured. The unchangeable area A22does not change in a 3D model. The areas A12and A22are thus not to be focused by the analyst. The overlapping portion of the changeable areas A11and A21is determined to be overlapping as a result of the device30operating as expected, and thus is not the area to be focused by the analyst.

In contrast, the area A30corresponds to an area in which the device30is not operating as intended or an area in which a substance other than a substance depicted as a 3D model is captured in the captured moving image. The area A30may possibly be associated with an abnormality and is thus to be focused by the analyst. The area A11corresponds to an example of a first area changeable in the captured moving image. The area A21corresponds to an example of a second area changeable in the model moving image.

The display controller15emphasizes, as illustrated with dark hatching inFIG.14, a portion extracted as described above as the area A30based on sequences of frame images. More specifically, the display controller15displays the captured moving image in a manner superimposed on the model moving image with the portion in the captured moving image corresponding to the area A30with lower transparency than the other portions. The area A30may be emphasized with a blinking dashed line indicating the area A30.

As described above, the display controller15corresponds to an example of the display data generation means for generating, based on a comparison between the first area changeable in the captured moving image and a second area changeable in the model moving image, the display data for partially emphasizing the first area.

Although the area11is partially emphasized in the above example, the area changeable in the captured moving image may be fully emphasized for potential changes in the captured moving image different from changes in the model moving image. A change in each pixel in the captured moving image and a change in each pixel in the model moving image may be compared, and a set of pixels with a difference between the changes greater than a predetermined threshold may be extracted as the area A30.

Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above embodiments.

For example, although the captured moving image is superimposed on the model moving image in the above examples, the model moving image may be superimposed on the captured moving image.

Instead of displaying the model moving image and the captured moving image in a superimposed manner, the model moving image and the captured moving image may be synchronously played to appear adjacent to each other as illustrated inFIG.15.

Although the imaging data indicates the capturing date and time of each frame image in the captured moving image, and the log data indicates the recording date and time of each record in the above examples, this is not limitative. For example, the imaging data may indicate the time length from when the controller31is activated to when each frame image is captured, and the log data may indicate the time length from when the controller31is activated to when each record is taken. The capturing time of each frame image indicated by the imaging data and the recording time of each record indicated by the log data may be times measured based on the same reference.

Although the model data indicates the object positions in addition to the third-dimensional model in the above examples, the object positions may be acquired separately from the model data by the model acquirer12.

Although the estimator14detects the objects in the captured moving image, the user may specify the positions of the objects in the captured moving image on the image.

Although the three markers used as objects in the above examples, four or more markers may be used. An object different from the marker may also be used. For example, a single mark with a two-dimensional pattern printed as an object as illustrated in the upper portion ofFIG.16may be attached to the support base, and the position and the orientation of the imager20may be estimated from the deformed state of the captured object, as illustrated in the lower portion ofFIG.16.

The display data generation apparatus10may include, in place of the display controller15, a display data generator15athat outputs the display data to an external display device160as illustrated inFIG.17. For example, the display data generation apparatus10being a cloud server on the Internet may distribute, to the display device160being a user terminal, a web page as display data containing the model moving image and captured moving image and a script for synchronously playing the captured moving image along with the model moving image. In the example inFIG.17, the display data generator15acorresponds to an example of the display data generation means.

The functions of the display data generation apparatus10may be implementable by dedicated hardware or a common computer system.

For example, the program48executable by the processor41may be stored in a non-transitory computer-readable recording medium for distribution. The program48may then be installed in a computer to provide a device that performs the above processing. Examples of such a non-transitory recording medium include a flexible disk, a compact disc ROM (CD-ROM), a digital versatile disc (DVD), and a magneto-optical (MO) disk.

The program48may be stored in a disk device included in a server device on a communication network such as the Internet, and may be, for example, superimposed on a carrier wave to be downloaded to a computer.

The above processing may also be performed by the program48being activated and executed while being transferred through a communication network.

The above processing may also be performed by entirely or partially executing the program48on a server device while a computer is transmitting and receiving information about the processing through a communication network.

In the system with the above functions implementable partially by the operating system (OS) or through cooperation between the OS and applications, portions executable by applications other than the OS may be stored in a non-transitory recording medium that may be distributed or may be downloaded to a computer.

Means for implementing the functions of the display data generation apparatus10is not limited to software. The functions may be partly or entirely implemented by dedicated hardware including circuits.

The foregoing describes some example embodiments for explanatory purposes. Although the foregoing discussion has presented specific embodiments, persons skilled in the art will recognize that changes may be made in form and detail without departing from the broader spirit and scope of the invention. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. This detailed description, therefore, is not to be taken in a limiting sense, and the scope of the invention is defined only by the included claims, along with the full range of equivalents to which such claims are entitled.

INDUSTRIAL APPLICABILITY

The structure according to one or more embodiments of the present disclosure may be used for analyzing abnormalities that occur at FA sites.

REFERENCE SIGNS LIST

100Display system10Display data generation apparatus11Moving image acquirer12Model acquirer13Log acquirer14Estimator15Display controller15aDisplay data generator16Display20Imager30Device31Controller41Processor42Main storage43Auxiliary storage44Input device45Output device46Communicator47Internal bus48Program101,102Moving image103Screen104Play button105Pause button160Display device300Workpiece301,303Tray302Inspector304Sensor