Information management apparatus, information management method, and non-transitory recording medium

An information management apparatus includes at least one processor configured to execute a program stored in a storage. The at least one processor acquires sequentially captured frames. The at least one processor acquires a movement state of a position of a light source in an imaging area, based on light source images contained in the acquired frames. The light source transmits information by means of light including an illumination pattern of the light. The at least one processor causes the storage to store the information and the movement state in association with each other.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of Japanese Patent Application No. 2020-019569, filed on Feb. 7, 2020, the entire disclosure of which is incorporated by reference herein.

FIELD

This application relates to an information management apparatus, an information management method, and a non-transitory recording medium.

BACKGROUND

In recent years, some systems have been devised that record moving images of operation sites (for example, factories and warehouses) and track and verify movements of people and objects for management purposes, thereby improving productivity and performing management of operations.

For example, Unexamined Japanese Patent Application Publication No. 2019-152802 discloses a technique that involves executing image analysis to acquire location information on body parts of an operator in a captured image and tracking behaviors of the body parts of the operator during a certain operation. This technique can achieve detection of an abnormal behavior in the operation on the basis of the image.

Unfortunately, the accuracy of detecting the positions of the body parts may be affected by the imaging environment in the above-mentioned technique.

SUMMARY

In order to solve the above problem, an information management apparatus according to a first aspect of the disclosure includes at least one processor configured to execute a program stored in a storage. The at least one processor acquires sequentially captured frames, acquires a movement state of a position of a light source in an imaging area based on light source images contained in the acquired frames, the light source transmitting information by means of light including an illumination pattern of the light, and causes the storage to store the information and the movement state in association with each other.

In order to solve the above problem, an information management method according to a second aspect of the disclosure, which is executed by an information management apparatus. The information management method includes acquiring frames sequentially captured by an imager, acquiring a movement state of a position of a light source in an imaging area based on light source images contained in the acquired frames, the light source transmitting information by means of light including an illumination pattern of the light; and causing a storage to store the information and the movement state in association with each other.

In order to solve the above problem, a non-transitory computer-readable recording medium according to a third aspect of the disclosure stores a program thereon executable by at least one processor of an information management apparatus. The program causes the at least one processor to acquire frames sequentially captured by an imager, acquire a movement state of a position of a light source in an imaging area based on light source images contained in the acquired frames, the light source transmitting information by means of light including an illumination pattern of the light, and cause a storage to store the information and the movement state in association with each other.

DETAILED DESCRIPTION

A visible light communication system as an information management system according to embodiments of the disclosure will now be described with reference to the accompanying drawings.

FIG.1illustrates an exemplary configuration of a visible light communication system. A visible light communication system600illustrated inFIG.1generates information corresponding to behaviors of an operator150. The visible light communication system600includes markers 1, 2, 10, 11, and 101 (hereinafter referred to collectively as “marker 1 and the like” as appropriate), a camera200, a server300, and a database400.

In this embodiment, each of the marker 1 and the like (light source) includes a light emitting diode (LED) (not shown) as an illumination device. The camera200is connected to the server300.

The marker 1 is mounted on a right hand151of the operator150, and the marker 2 is mounted on a left hand152of the operator150. The marker 10 is mounted on a tool161used by the operator150, and the marker 11 is mounted on a tool162used by the operator150. The marker 101 is mounted on a tray171on which parts (not shown) are placed thereon.

The LED in each of the marker 1 and the like transmits an identification (ID) that is information for uniquely identifying the marker 1 or the like including the LED by emitting a light modulated by means of a variation in hue with time. In the embodiment, the marker 1 has an ID of 1, the marker 2 has an ID of 2, the marker 10 has an ID of 10, the marker 11 has an ID of 11, and the marker 101 has an ID of 101.

The camera200captures an image of the space encompassing the marker 1 and the like. The server300acquires information, such as the IDs of the marker 1 and the like, from the image of lights (light source image) captured by the camera200.

FIG.2illustrates exemplary configurations of the camera200, the server300, and the database400. The camera200and the server300constitute an information management apparatus. As illustrated inFIG.2, the camera200includes an imager202and a lens203. The server300includes a control unit302, an image processor304, a memory305, an operation unit306, a display307, and a communicator308.

The lens203in the camera200is a zoom lens, or the like. The lens203is shifted in response to a zooming control operation from the operation unit306in the server300and a focusing control by the control unit302. The shift of the lens203controls the angle of view of the imager202and optical images captured by the imager202.

The imager202is equipped with multiple light receiving elements arranged in a regular two dimensional array and the multiple light receiving elements form a light receiving surface including an imaging surface. The light receiving elements are each an imaging device, such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The imager202performs successive capturing (light receiving) of optical images incident through the lens203in a predetermined range of angle of view, in accordance with a control signal from the control unit302in the server300. The imager202then converts image signals within the angle of view into digital data to generate frames (captured images). The imager202captures an image and generates an image frame sequentially in time, and outputs the successive frames to the image processor304.

The image processor304outputs digital data on the frames output from the imager202, to the control unit302, in accordance with a control signal from the control unit302.

The control unit302includes a central processing unit (CPU), for example. The control unit302executes software processes in accordance with programs stored in the memory305, and thereby controls various functions of the server300, such as a function of conducting the operations explained later.

The memory305includes, for example, a random access memory (RAM) and a read only memory (ROM). The memory305stores various information, such as programs, to be used in controls and the like in the server300, and the digital data on the frames output from the image processor304to the control unit302.

The operation unit306includes a numeric keypad, function keys, and the like and serves as an interface for input of an operation of a user. The display307is, for example, a liquid crystal display (LCD), a plasma display panel (PDP), or an electroluminescence (EL) display. The display307displays an image in accordance with the image signal output by the control unit302. The communicator308includes, for example, a local area network (LAN) card. The communicator308communicates with external communication apparatuses.

The control unit302is configured by a position acquirer334, a storage controller336, and a reproduction controller338.

The position acquirer334acquires the positions corresponding to the marker 1 and the like in each of the images (hereinafter referred to as “frames”) continuously output from the image processor304in the chronological order. For example, the position acquirer334determines the sites having a luminance value equal to or greater than a predetermined value in the frame, as the positions corresponding to the marker 1 and the like in the frame.

The storage controller336detects IDs associated with the illumination patterns (variations in luminance and color) at the positions of the marker 1 and the like in the frame, and calculates movement vectors indicating variations with time of the positions corresponding to the marker 1 and the like.

Specifically, the storage controller336specifies the positions of each marker corresponding to the identical ID in the current frame and the last frame among the frames continuously output from the image processor304in the chronological order. The storage controller336then calculates a length and angle of the movement vector, starting at the position of the marker in the last frame and ending at the position of the marker in the current frame. The calculated length and angle indicate a displacement of the marker corresponding to the identical ID. In the case where the marker corresponding to the identical ID resides at the same position in both of the current frame and the last frame, the movement vector is calculated as 0. In response to every output of a new frame from the image processor304to the control unit302, the lengths and angles of the movement vectors of the markers in the frame are calculated as explained above.

The storage controller336then compares the latest movement vector and the last movement vector of the identical marker and determines whether the variation in at least one of the length and angle exceeds a predetermined threshold. If the variation in at least one of the length and angle is determined to exceed the predetermined threshold, the storage controller336generates basic index information indicating a movement of the marker.

An exemplary generation of basic index information will now be explained.FIG.3Aillustrates exemplary operation process executed by a certain operator150, andFIG.3Billustrates an exemplary basic index table401, which is a collection of basic index information corresponding to behaviors of the operator150. The basic index table401is stored in the database400as explained later.

As illustrated inFIG.3A, the operation process executed by the operator150involves:

(1) causing the right hand151to approach the tray171for picking up a part;

(2) causing the right hand151to bring the part;

(3) causing the left hand152to approach the tool161for holding the tool161;

(4) causing the left hand152to bring the tool161; and

(5) processing the part with the tool161,

executed by the operator150.

The operation of “(1) causing the right hand151to approach the tray171for picking up a part” inFIG.3Ais defined as a mode in which the position of ID=1 (at which the marker 1 mounted on the right hand151emits light and transmits information) starts to move from a resting state. This mode is managed as Index No. 1 in the basic index table401illustrated inFIG.3B. Index No. 1 is associated with the frame number “30”, which indicates the time point when a variation in at least one of the length and angle of the movement vector of ID=1 exceeds a first threshold. Index No. 1 is also associated with meta-information that “ID=1 (that is, the right hand151) starts to move”. Specifically, the first threshold is defined assuming at least one of the case in which the length of the movement vector is a predetermined length or longer (that is, the moving speed between frames is a predetermined speed or higher) and the case in which the angle of the movement vector is inverted (specifically, the variation in the angle is approximately 150° or larger).

The operation of “(2) causing the right hand151to bring the part” inFIG.3Ais defined as a mode in which the angle of the movement vector of ID=1 is varied (specifically, the moving direction is substantially inverted). This mode is managed as Index No. 2 in the basic index table401illustrated inFIG.3B. Index No. 2 is associated with the frame number “45”, which indicates the time point when a variation in the angle of the movement vector of ID=1 exceeds the first threshold due to inversion of the moving direction. Index No. 2 is also associated with meta-information that “the movement vector of ID=1 (that is, the right hand151) is varied”.

The operation of “(3) causing the left hand152to approach the tool161for holding the tool161” inFIG.3Ais defined as a mode in which while the position of ID=1 is in a resting state, the position of ID=2 (at which the marker 2 mounted on the left hand152emits light and transmits information) starts to move from a resting state, and then the position of ID=2 and the position of ID=10 (at which the marker 10 mounted on the tool161emits light and transmits information) reside within a predetermined area. This mode is managed as Index No. 3 in the basic index table401illustrated inFIG.3B. Index No. 3 is associated with the frame number “224”, which indicates the time point when a variation in at least one of the length and angle of the movement vector of ID=2 exceeds a first threshold while the position of ID=1 is constant and then the position of ID=2 and the position of ID=10 enter the predetermined area. Index No. 3 is also associated with meta-information that “ID=1 (that is, the right hand151) is in a resting state and ID=2 (that is, the left hand152) starts to move”.

The operation of “(4) causing the left hand152to bring the tool161” inFIG.3Ais defined as a mode in which the angle of the movement vector of ID=2 is varied (specifically, the moving direction is inverted) and the position of ID=10 starts to move from a resting state together with the position of ID=2 at the same timing. This mode is managed as Index No. 4 in the basic index table401illustrated inFIG.3B. Index No. 4 is associated with the frame number “239”, which indicates the time point when the angle of the movement vector of ID=2 is varied due to inversion of the moving direction and the positions of ID=2 and ID=10 start to move together. Index No. 4 is also associated with meta-information that “the movement vector of ID=2 (that is, the left hand152) is varied and ID=10 (that is, the tool161) starts to move”.

The operation of “(5) processing the part with the tool161” inFIG.3Ais defined as a mode in which the positions of ID=1, 2, and 10 move within the predetermined area. This mode is managed as Index No. 5 in the basic index table401illustrated inFIG.3B. Index No. 5 is associated with the frame number “410”, which indicates the time point when the positions of ID=1, 2, and 10 reside within the predetermined area and variations in (lengths and angles of) the movement vectors are smaller than a predetermined second threshold. Index No. 5 is also associated with meta-information that “ID=1 (that is, the right hand151), ID=2 (that is, the left hand152), and ID=10 (that is, the tool161) move within the predetermined area”. Specifically, the second threshold is defined assuming the case in which the lengths of the movement vectors are shorter than the predetermined length defined by the first threshold (that is, the moving speed between frames is at most the predetermined speed defined by the second threshold).

The operation of “(6) completing the operation” inFIG.3Ais defined as a mode in which all the IDs in the imaging area are in a resting state. This mode is managed as Index No. 6 in the basic index table401illustrated inFIG.3B. Index No. 6 is associated with the frame number “524”, which indicates the time point when variation cannot be found at the positions of ID=1, 2, 10, 11, and 101, and associated with meta-information that “all the IDs are in a resting state”.

The operations of generating the basic index table401will now be explained.FIG.4is a flowchart illustrating the entire operations executed by the camera200and the server300.FIG.5is a flowchart illustrating the operation of the server300among the entire operations.

First, the imager202in the camera200conducts successive image capturing and sequentially generates frames (Step S101).

The control unit302of the server300then generates the basic index table401on the basis of movements of the markers in the frames and the IDs of the markers (Step S102).

Step S102will be explained in more detail. As illustrated inFIG.5, the position acquirer334in the control unit302of the server300acquires the positions of the markers in the current frame (latest frame) (Step S201).

The storage controller336in the control unit302then detects IDs associated with the illumination patterns at the positions of the markers in the frames (Step S202).

The storage controller336then calculates a movement vector of each marker on the basis of a displacement of the marker corresponding to the detected identical ID between the current frame and the last frame (precedent frame) (Step S203).

The storage controller336then compares the latest movement vector and the last movement vector corresponding to the identical ID and determines whether any movement vector has a variation in at least one of the length and angle that is equal to or greater than the first threshold (Step S204).

If variations in both of the lengths and angles of all the movement vectors are determined to be smaller than the first threshold (Step S204; NO), the storage controller336compares the latest movement vector and the last movement vector corresponding to the identical ID and determines whether any movement vector has a variation in the length that is smaller than the second threshold (Step S205). If any movement vector is determined to have a variation in the length that is smaller than the second threshold (Step S205; YES), the storage controller336determines whether the number of the movement vectors whose variation in the length is smaller than the second threshold satisfies a predetermined number (Step S206). The storage controller336then generates basic index information corresponding to the ID of the movement vector that satisfies the conditions of Steps S204to S206and adds the generated basic index information to the basic index table401(Step S207). In contrast, if any of the conditions of Steps S204to S206are not satisfied, the process returns to Step S201.

It should be noted that the above-explained conditions of the length and angle of the movement vector required for addition of information to the basic index table401are mere examples and not limited thereto. For example, the conditions required for addition of information to the basic index table401involve the case in which the lengths of all the movement vectors are maintained to be 0 for a certain period, which is deemed as a mode of “completing the operation”, the case in which some IDs gather within the predetermined area or move away from each other, and other various cases of movement of IDs.

Referring back toFIG.4, the storage controller336generates moving image data containing sequentially captured frames, converts the data into a file, stores the file into a moving image data storage405of the database400, and also stores the generated basic index table401into the database400(Step S103). The moving image data and the basic index table401are stored in association with each other.

Referring back toFIG.2, the above-explained process of generating and storing the basic index table401is followed by the process explained below. The reproduction controller338in the control unit302causes the information in the basic index table401to be displayed on the display307.

While checking out the content in the basic index table401displayed on the display307, a user designates a frame number that the user desires to reproduce in the moving image data corresponding to the basic index table401.

In response to designation of the frame number to be reproduced, the reproduction controller338reads the moving image data stored in the moving image data storage405, reads the frame corresponding to the designated frame number from the moving image data, and then causes the frame to be reproduced and displayed on the display307. The user can thus reproduce the moving image containing the designated frame and the following frames of the moving image data.

The user also executes an operation of generating an element definition table402with reference to the basic index table401and the moving image data. The element definition table402specifies the existing areas of the objects (elements) provided with the markers 1 and the like corresponding to the IDs in the frames.

FIG.6illustrates an exemplary element definition table402. As illustrated inFIG.6, the element definition table402, which is generated for each ID, contains element numbers, IDs, the names of the objects (elements) provided with the marker 1 and the like corresponding to the IDs, and existing areas (coordinates of the four corners of the rectangular areas) of the elements in the frames. In this embodiment, the existing area is set only for the tray171. Since the existing area is set for the tray171, the movement of the marker 1 into this existing area of the tray171represents that the element name “right hand151” approaches the element name “tray171” to pick up a part, for example. Also, the movement of the marker 1 that has been in the existing area of the tray171out of the existing area represents that the element name “right hand151” picks up a part out of the element name “tray171”. The element definition table402generated as explained above is stored into the database400.

The user also executes an operation of generating a relation definition table403with reference to the basic index table401and the moving image data. The relation definition table403illustrates positional relationships of the objects (elements) provided with the marker 1 and the like corresponding to the IDs.

FIG.7illustrates an exemplary relation definition table403. As illustrated inFIG.7, the relation definition table403contains “elements” (elements 1 and 2 in this example) identified by the IDs of the respective markers, “expression of positional relation” that represents elements and a condition of positional relationship between the elements, “operation name”, and “operation type”. InFIG.7, the expression “element 1 into element 2” indicates that the position of “element 1” becomes overlapped with “predetermined area centered around the position of the element 2”. In this case, “element 1” and “element 2” are connected by the term “into”. The expression “element 1 out from element 2” indicates that “element 1” that has been within “predetermined area centered around the position of the element 2” moves “away from” the predetermined area. In this case, “element 1” and “element 2” are connected by the term “out from”. It should be noted that the terms “into” and “out from” are used provided that either of the elements 1 and 2 has an existing area illustrated inFIG.6. The expression “element 1 hidden element 2” indicates that the position of “element 1” becomes hidden behind “predetermined area centered around the position of the element 2”. Specifically, the expression indicates that the position of the ID corresponding to “element 1” moves to the vicinity of the position of the ID corresponding to “element 2” and then becomes hidden behind “element 2” and cannot be specified. The expression “element 1 with element 2” indicates that the positions of “element 1” and “element 2” reside within the predetermined area and their movement vectors have approximately the same lengths and angles, that is, “element 1” and “element 2” perform the same movement. The expression “element 1 uncover element 2”, which is not listed inFIG.7, indicates that “element 1” that has been hidden behind “element 2” appears again. The operation type is information for classifying an operation of the entire operation process as “preparing operation” or “main operation”. The “main operation” indicates an operation to be mainly a check target in the entire operation process, and the “preparing operation” indicates a phase of preparation of the “main operation”. The relation definition table403generated as explained above is stored into the database400.

The user also executes an operation of generating a behavior series table404with reference to the basic index table401and the moving image data. The behavior series table404contains the operation names of the respective operation processes executed by the operator150and recorded in the moving image data, in association with the frame numbers of the moving image data.

FIG.8illustrates an exemplary behavior series table404. As illustrated inFIG.8, the behavior series table404contains indexes set for the individual process names, frame numbers of the frames of the moving image data that record the processes associated with the process names, and process names to be subject to conformity determination. The generated behavior series table404is stored into the database400.

The above-explained tables (the basic index table401, the element definition table402, the relation definition table403, and the behavior series table404) serving as reference information tables are generated and stored into the database400. Then, another operator, whose right hand151is provided with the marker 1 transmitting ID=1 and left hand152is provided with the marker 2 transmitting ID=2, performs the operations in the above-explained operation process. A moving image of this operator is captured at the same angle as in the capturing of the moving image data associated with the generated tables. On the basis of the positions of the markers (markers 1, 2, 10, 11, and 101) in the frames sequentially captured and output, the variations in the lengths and angles of the movement vectors are acquired and compared with the contents of the tables, so as to determine whether the current operator succeeds to perform the processes in the operation process in accordance with the contents of the tables. Specifically, the control unit302reads the tables401to404corresponding to an operation process to be evaluated from the database400. The control unit302then refers to the read tables and determines whether the behavior of the current operator matches the reference behavior in the operation process, or whether the initial frame associated with the process name is identical to the reference initial frame or within a predetermined allowable range, on the basis of the variations in the lengths and angles of the movement vectors of the markers in the frames sequentially captured and output. If determining that the initial frame is identical or within the allowable range, the control unit302determines no problem in the behavior in the operation process by the current operator. If determining that the initial frame is different or out of the allowable range in any behavior in the operation process, the control unit302provides an alert screen or notification at this timing for informing the user that the behavior does not matches the reference behavior, for example.

The above description of the embodiment and the drawings should not be construed as limiting the disclosure and may be modified as appropriate.

For example, although the marker 1 and the like transmit only IDs in the above embodiment, the marker 1 and the like may transmit other information.

FIG.9Ais a circuit configuration diagram of another transmission apparatus500, andFIG.9Billustrates an appearance diagram of the other transmission apparatus500. The transmission apparatus500illustrated inFIG.9Aincludes a marker502, a control unit503, a memory504, sensors506ato506e, a communicator508, a driver512, and a battery550. As shown in the appearance diagram ofFIG.9B, the transmission apparatus500has a glove-like shape and is provided with the sensors506ato506efor detecting extension and contraction at the respective positions corresponding to the five fingers. The transmission apparatus500is also provided with the marker502at the position corresponding to the back of the hand. The marker502is mounted on the hand of an operator that mainly performs behaviors.

Referring back toFIG.9A, the control unit503includes a CPU, for example. The control unit503executes software processes in accordance with programs stored in the memory504, and thereby controls various functions of the transmission apparatus500.

The memory504includes, for example, a RAM and a ROM. The memory504stores various information, such as programs, to be used in controls and the like in the transmission apparatus500. The sensors506ato506edetect movements of the fingers of the operator. The communicator508is a wireless communicator and communicates with other communication devices, such as the server300. The battery550supplies the individual components with electric power necessary for operations of the transmission apparatus500.

The control unit503is configured by an illumination controller524. The illumination controller524determines an illumination pattern of light to be emitted from the marker502, in association with the ID of the transmission apparatus500and information on the movements of the fingers of the operator detected by the sensors506ato506e.

In addition, the illumination controller524outputs information on the illumination pattern associated with the ID and movements of the fingers to the driver512. On the basis of the information on the illumination pattern from the illumination controller524, the driver512generates a driving signal for causing a variation with time of the light to be emitted from the marker502. The marker502emits light in accordance with the driving signal output from the driver512.

Because of this transmission apparatus500, the server300can generate a more detailed basic index table401that also reflects movements of the fingers of the operator.

FIG.10Ais a circuit configuration diagram of still another transmission apparatus501, andFIG.10Billustrates an appearance diagram of the other transmission apparatus501. The transmission apparatus501illustrated inFIGS.10A and10Bis equipped with a camera507instead of the sensors506of the transmission apparatus500illustrated inFIG.9.

As illustrated inFIG.10B, the transmission apparatus501is a moving body, such as a forklift. The forklift travels when a travel controller510is controlled by operation of an operation unit509, for example, and carries a load511. The load511is labelled with a barcode513for identifying the type of the load511. The camera507captures an image of the barcode513.

Referring back toFIG.10A, the illumination controller524in the control unit503determines an illumination pattern of light to be emitted from the marker502, in association with the ID of the transmission apparatus501and information on the type of the load511indicated by the barcode513in the image captured by the camera507.

In addition, the illumination controller524outputs information on the illumination pattern associated with the ID and the information on the type of the load511to the driver512. On the basis of the information on the illumination pattern from the illumination controller524, the driver512generates a driving signal for causing a variation with time of the light to be emitted from the marker502. The marker502emits light in accordance with the driving signal output from the driver512.

Because of this transmission apparatus501, the server300can generate a more detailed basic index table401that also reflects the type of the load511being carried.

Although the above embodiments include no limitation regarding visible light, the communication may be performed using red, green, and blue lights or lights of other colors. The disclosure can also be applied to visible light communication in which information is modulated by means of only a variation in luminance with time.

The information transmitted from the marker 1 and the like may also be information on an error in devices provided with the marker 1 and the like, for example, other than the IDs, movements of fingers, and type of a load in the above embodiments.

The light sources in the marker 1 and the like should not necessarily be LEDs. For example, a part of the LCD, PDP, or EL display constituting the display may function as light sources.

The server300may be equipped with the camera200therein.

In the above embodiments, the program to be executed may be stored for distribution in a non-transitory computer-readable recording medium, such as a flexible disk, a compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), or a magneto-optical (MO) disc. In this case, this program is installed into a computer to configure a system executing the above-explained operations.

Alternatively, the program may be stored in a disk drive or the like included in a certain server on a network, such as the Internet, and may be downloaded into a computer, for example, by superimposing the program onto carrier waves.

If the above functions are shared by an operating system (OS) or achieved by cooperation between the OS and application, only the data other than the OS may be stored in a medium for distribution or downloaded into a computer, for example.