Apparatus and method for information processing and storage medium

An apparatus includes an acquisition unit configured to obtain a signal indicating an address of a pixel in which a change in luminance has occurred and time of the change and a generation unit configured to generate a time-series image indicating a position of at least one pixel in which a change in luminance has occurred and a direction of the change in luminance based on the signal, wherein, when the time-series image is to be displayed in reverse chronological order, the generation unit generates the time-series image in which the direction of the change in luminance is reversed.

BACKGROUND OF THE DISCLOSURE

Field of the Disclosure

The aspect of the embodiments relates to display of event base sensors.

Description of the Related Art

There is a known event base sensor that outputs a change in luminance of each pixel in real time using an address event signal (see Japanese Patent Laid-Open No. 2019-134271).

SUMMARY OF THE DISCLOSURE

An apparatus according to an aspect of the embodiments includes an acquisition unit configured to obtain a signal indicating an address of a pixel in which a change in luminance has occurred and time of the change and a generation unit configured to generate a time-series image indicating a position of at least one pixel in which a change in luminance has occurred and a direction of the change in luminance based on the signal, wherein, when the time-series image is to be displayed in reverse chronological order, the generation unit generates the time-series image in which the direction of the change in luminance is reversed.

DESCRIPTION OF THE EMBODIMENTS

Information processing apparatuses according to embodiments of the disclosure will be described hereinbelow with reference to the drawings. Components having the same function in all the drawings are denoted by the same reference signs, and descriptions thereof will be omitted.

First Embodiment

The following is a description on an information processing apparatus including an event base sensor that displays an image generated from an address event signal indicating the address of a pixel where a change in luminance has occurred and the time of the change. In particular, a method of display using a three-dimensional coordinate space constituted by the coordinates (XY) of a pixel in which an event has occurred and an axis (T) indicating the time (T) when a change in luminance has occurred (hereinafter referred to as “XYT display”) will be described.

FIG.1is a block diagram of an example of the hardware configuration of the information processing apparatus. InFIG.1, the information processing apparatus100includes an image capturing unit101including an imaging optical system1010and a photoelectric conversion element1011, a central processing unit (CPU)102, a memory103, and a display unit104. The image capturing unit101outputs an address event signal according to received incident light. The imaging optical system1010is specifically a light receiving lens, and receives incident light and forms an image on the photoelectric conversion element1011. The photoelectric conversion element1011is specifically a single photon avalanche diode (SPAD) sensor or a complementary metal oxide semiconductor (CMOS) sensor and outputs an address event signal according to the received incident light. The CPU102reads an operating system (OS) or other programs stored in the memory103and executes them to control connected components to perform, for example, calculation of various processes and logical determination. The processes that the CPU102executes include information processing according to this embodiment. The CPU102also controls the focusing and aperture driving of the imaging optical system1010and driving of the photoelectric conversion element1011. Examples of the memory103include a hard disk drive and an external storage, which store programs and various data on the information processing of the embodiment. The display unit104performs XYT display according to various information and the address event signal.

Referring next toFIG.2, an example of the functional configuration of the information processing apparatus100according to this embodiment will be described. The information processing apparatus100includes the image capturing unit101, an acquisition unit201, an image generation unit202, a display control unit203, the display unit104, and a user interface (an operation unit)105. The image capturing unit101outputs an address event signal according to the received incident light. More specifically, the image capturing unit101outputs an address event signal indicating the address of a pixel where a change in luminance occurred and the time of the change. The acquisition unit201obtains the address event signal output from the event base sensor. The address event signal includes information on the position of a pixel (or a set of pixels) where a change in luminance occurred in a predetermined time range, the direction of the change in luminance, and the time when the change (event) in luminance occurred. The image generation unit202generate a time-series image indicating the position of at least one pixel where a change in luminance occurred in a predetermined time range and the direction of the change in luminance. In other words, the image generation unit202generates an XYT image from the address event signal. When displaying the image in reverse chronological order, the display control unit203reverses the direction of the change in luminance of the generated image. When displaying the image in reverse chronological order, the display unit104displays an image in which the direction of the change in luminance is reversed. For example, the display unit104may be a device outside the information processing apparatus100, for example, a display or a projector. The user interface105receives various instructions on image capturing, display, and so on input by the user. The user interface105may be an external device separate from the information processing apparatus100. The functional configuration of the information processing apparatus100may include a configuration other than those described here.

An example of the event base sensor according to this embodiment will be described. The event base sensor counts the number of incident photons and determines the timing at which the number of the counted photons exceeded a predetermined threshold. The event base sensor measures the time (clocks) required until the number of photons reached a first threshold or greater and detects a change in luminance by comparing the required times. Specifically, if the difference T−T0between the last-measured required time T0and the latest required time T is a second threshold or greater, the event base sensor detects a change in luminance in the minus direction. If the difference T0−T is the second threshold or greater, the event base sensor detects a change in luminance in the plus direction. If the difference between T and T0is less than the second threshold, the event base sensor does not detect a change in luminance. The second threshold is a value equal to or greater than zero, for example, a preset value or a value that is set according to another parameter. The detailed configuration will be described hereinbelow.FIG.3Ais a diagram illustrating an example configuration of the photoelectric conversion element1011.

The photoelectric conversion element1011includes a pixel unit110and a peripheral circuit120. The peripheral circuit120includes a vertical adjustment circuit121and a horizontal readout circuit122.

FIG.3Bis a diagram illustrating an example configuration of each pixel unit110of the event base sensor. The pixel unit110includes a photoelectric conversion unit111, a pixel counter112, a time counter113, a first determination circuit114, a memory115, a comparator116, a second determination circuit117, a response circuit118, and a selection circuit119.

The photoelectric conversion unit111includes a single photon avalanche diode (SPAD) that operates in Geiger mode and counts the number of photons incident on the photoelectric conversion unit111with the pixel counter112. The time counter113counts the time when photons are incident on the photoelectric conversion unit111. The event base sensor including the SPAD can detect a change in luminance in the order of a photon. Detecting a change in luminance per photon allows an address event signal to be given even in night vision, such as at nighttime.

When the number of photons counted by the pixel counter112reaches the first threshold, the first determination circuit114stops the time count with the time counter113. The memory115stores past values counted by the time counter113. The difference between the present count value of the time counter113and the past count value of the time counter113is obtained with the comparator116.

If the value of the difference is the second threshold or greater, the second determination circuit117sends a request signal to the vertical adjustment circuit121via the response circuit118. The response circuit118receives a response indicating whether to permit output of address event data from the vertical adjustment circuit121. If the difference count value is less than the second threshold, the second determination circuit117sends no request signal.

When the response circuit118receives a response indicating permission of the output, the selection circuit119outputs the count value of the time counter113to the horizontal readout circuit122. The horizontal readout circuit122outputs the received count value as an output signal to a detection unit from the photoelectric conversion element1011.

Since the differential count value calculated by the comparator116corresponds to the reciprocal of the photon incident frequency, the photoelectric conversion element1011of the disclosure has the function of measuring “a change in photon incident frequency”, that is, a change in luminance. Only when the difference in the interval of the time when the number of incident photons reached the first threshold, determined by the second determination circuit117, is the second threshold or greater, the photoelectric conversion element1011outputs the address event signal. In other words, if the difference in incident frequency is large, the photoelectric conversion element1011outputs the incident frequency, and if the difference is small, the photoelectric conversion element1011does not output the incident frequency. The above configuration provides an asynchronous photoelectric conversion element that detects a change in luminance as an address event for each pixel address in real time.

The foregoing illustrates a photoelectric conversion element that detects a change in photon incident frequency by measuring the time when photons enter using the SPAD for the photoelectric conversion unit111. However, the configuration inFIG.2is given for mere illustrative purposes. Any asynchronous photoelectric conversion element that detects changes in luminance as address events in real time may be used. For example, a photoelectric conversion element that detects a change in luminance as a voltage change may be used, as in Japanese Patent Laid-Open No. 2019-134271.

The image generation unit202generates a time-series image indicating the position of at least one pixel where a change in luminance occurred in a predetermined time range and the direction of the change in luminance according to a given address event signal. Specifically, the image generation unit202rearranges the information on the luminance change detected by the event base sensor to a raster scan format, which is easy for the display unit104to display. In other words, the image generation unit202generate an XYZ image by converting the address event signal to a coordinate space constituted by T-axis (first axis) indicating the time axis and XY axes (second axes) related to space.

The order of signal output from the photoelectric conversion element1011is controlled by the vertical adjustment circuit121in the photoelectric conversion element1011, as described above. The signals are output in the order in which a change in luminance equal to or greater than the threshold occurred (in time-series). For that reason, unlike synchronous photoelectric conversion elements, such as common CMOS image sensors, the order of output from the pixels is not in the raster-scan frame format. The image generation unit202therefore temporarily stores signals output from the photoelectric conversion element1011in a specific time range in the memory103and thereafter rearranges the signals for each pixel address to convert them into a raster scan frame format. Repeating this conversion multiple times allows for generating an XYT image having multiple frame images (XY images) in the temporal direction (T-direction).

If the phenomenon is to be displayed in real time, the reciprocal of the refresh rate of the display unit104is used for the specific time range. If a high-speed change of the subject is to be displayed in slow motion, the specific time is decreased according to the display speed. Specification from the user may be input and set via the user interface105.

The display unit104displays the coordinates of the pixel where a change in luminance occurred and the time of change in three dimensions.FIG.4Aillustrates an example of the XYT image displayed by the display unit104. White indicates that the luminance has changed in the plus direction, black indicates that the luminance has changed in the minus direction, and gray indicates that the luminance has not changed (or less than a threshold).

Processing that the information processing apparatus100executes to display the address event signal described above will be described.FIG.10is a flowchart illustrating the processing executed by the information processing apparatus100. The processing illustrated in the flowchart ofFIG.10is executed by the CPU102, which is a computer, according to a computer program stored in the memory103. Expression of the operations (steps) is omitted in the following description by expressing each step as S at its head. The information processing apparatus100does not need to perform all of the steps of the flowchart.

At S1001, the information processing apparatus100initializes various settings. For example, the information processing apparatus100sets the time range when measuring the number of times of occurrence of a change in luminance. The information processing apparatus100also sets a threshold according to photon count noise. For example, the information processing apparatus100obtains the result of measurement of a change in luminance, without moving object in the monitored area to obtain a values at which an address event signal is observed. Since the observed address event signal is likely to be noise, the number of times of generation of noise is set as a threshold. Another method for setting the threshold may be employed. Any value may be set by the user. Next at S1002, the acquisition unit201obtains the address event signal output from the event base sensor. The address event signal includes the address of a pixel where a change in luminance occurred, the direction of the change in luminance, and time information with which the time when the luminance changed can be specified. At S1003, the display control unit203determines whether to display the address event signal in chronological order or in reverse chronological order. Here, the determination is made on the basis of whether the display method input to the user interface105by the user is a method in chronological order. The determination on the display method may be based on a preset display method.

For example, the time series may be switched between the forward direction and the reverse direction every predetermined time. If the display control unit203determines to display the address event signal in chronological order, the processing proceeds to S1005, and if not, the processing proceeds to S1004. At S1004, the image generation unit202generates a reverse time-series image indicating the position of at least one pixel where a change in luminance occurred and the direction of the change in luminance. In this case, to display the image in reverse chronological order, the image generation unit202generates an image in which the direction of the change in luminance that the obtained address event signal indicates is reversed. When address event signals are accumulated to some extent, the image generation unit202adds up changes in luminance in a specific time range for each pixel. For a change in luminance, the direction of the change in the plus direction or the minus direction and the amount of change are determined. For reverse chronological display, the image generation unit202generates an image after converting the direction of the change in luminance, from the plus direction to the minus direction or from the minus direction to the plus direction, with the amount of change in luminance kept. At S1005, the image generation unit202generates a time-series image indicating the position of at least one pixel where a change in luminance occurred and the direction of the change in luminance on the basis of the address event signal. In this case, for display in the forward direction on the time axis, the image generation unit202generates the image, with the direction of the change in luminance unchanged. At S1006, the display control unit203displays the generated image on the display unit104in a set time-series. At S1007, the information processing apparatus100determines whether to terminate the processing. The determination of the termination may be made on the basis of user's termination instruction or depending on whether all preset processing objects have been processed. If No, the processing returns to S1002.

The user interface105is used by the user to control the information processing apparatus100. Specifically, the user interface105has functions of switching the look-down direction of the XYT image as inFIG.4B, scaling up and down the axes of the XYT image, and clipping and displaying part of the XYT image. In particular, the user interface105of the disclosure has the function of switching the playback direction of the XYT image displayed on the display unit104between the forward direction (the direction in which time goes) and the reverse direction (the direction in which time goes back).FIG.11illustrates an example of the user interface105. Reference sign G1100denotes a graphic user interface (GUI). The user can control display of the address event signal on the screen of the GUI using a touch panel or various input devices. The GUI displays “During Playback” or “During Reverse Playback” on the screen to show the user how the image is displayed. G1101denotes a screen that displays an image based on the address event signal according to this embodiment. In this case, the address event signal is plotted on the XYT-axes three-dimensional coordinate space. The image may be displayed with the axes rotated, or an image generated at each point in time may be displayed. G1102denotes a button for changing the time axis to the opposite side or this side of the viewpoint. G1103denotes a button for rotating the space axes XY. G1104denotes a button for adjusting the playback speed of the generated image. G1105denotes a button for initializing various settings. G1106denotes a user interface (UI) for specifying the time range of the displayed image. The user can specify the start point and the end point to generate an image. G1107denotes a reverse play button. This button is selected when an image is to be displayed, with the time axis reversed. G1108denotes a pause button. G1109denotes a play button. This button is selected when an image is to be displayed in the forward direction on the time axis. The GUI is not limited to the above. The display unit104and the user interface105may be provided at an external device connected to the information processing apparatus100with a network cable or radio transmission.

The information processing apparatus100according to this embodiment plays back an XYT image in the reverse direction, with the luminance change of each pixel reversed. This display increases the possibility that the user can correctly recognize the phenomenon occurring in a use case of detecting abnormalities of a subject moving periodically at high speed. For example,FIG.9Cillustrates an image obtained by rearranging the frame images inFIG.9B, which are forward playback images, in the reverse direction. The coordinates1030inFIG.9Bare displayed in black at time T0. This shows that the luminance changed in the minus direction at time T. When the actual change in luminance is played back in the reverse direction, the luminance changes in the plus direction by the threshold or greater at time T. However,FIG.9Cshows a phenomenon in which the luminance has changed by the threshold or greater in the minus direction because the coordinates1030are displayed in black at time T0. In other words, the change in luminance expressed inFIG.9Bis not played back correctly in the reverse direction. Accordingly, the direction of the change in luminance seems to be the same between the case of chronological playback of the image and the case of reverse chronological playback of the image. This causes a phenomenon in which a change different from the actual change is displayed during the reverse playback.

FIGS.4A and4Billustrate a forward playback image and a reverse playback image according to this embodiment, respectively.FIG.5Aillustrates a cross section of the XYT image inFIG.9Ataken along Y=0, as inFIG.9B.FIG.5Billustrates a reverse playback image obtained by converting the forward playback image inFIG.5Ausing the method of this embodiment.

The reverse playback image according to this embodiment inFIG.5Breversely displays the direction of the change in luminance of each frame inFIG.5A. In other words, the pixels expressed in white inFIG.5Aare converted to black inFIG.5B, and the pixels expressed in black inFIG.5Aare converted to while inFIG.5B.

Coordinates130inFIG.5A, which correspond to the coordinates1030inFIG.9B, will be discussed.FIG.5AandFIG.9Billustrate the same playback image. The coordinates130inFIG.5Aare expressed in black at time T0, which shows that the luminance has changed in the minus direction at time T0. In other words, if the actual change in luminance is played back in the reverse direction, the luminance will change in the plus direction at time T. In contrast, the coordinates130inFIG.5Bare expressed in white at time T0, which shows that the luminance has changed in the plus direction at time T0. In other words, the reverse playback image inFIG.5Bis a correct reverse playback image of the forward playback image inFIG.5A.

Whether Forward Playback or reverse playback may be displayed on the display using character information.

A conceivable example is a case in which an information processing apparatus including the event base sensor detects the occurrence of an abnormality of a subject that periodically moves at high speed, such as an assembly robot in a factory. In such a use case, the user specifies the time zone where an abnormality of the subject, such as disturbance of the periodic movement, may have occurred by viewing the XYT image. Next, to analyze the time zone where the abnormality may have occurred, the user determines whether an abnormality has occurred and specifies its cause while repeatedly playing back the XYT image in the forward direction (the direction in which time goes) and in the reverse direction (the direction in which time goes back). Thus, in playing back the XYT image in the reverse direction, the change in luminance of each pixel is reversely displayed. Such display increases the possibility that the user can correctly recognize the occurring phenomenon in a use case of detecting the abnormality of a subject that moves periodically at high speed.

Second Embodiment

<Changing Display at Reverse Playback>

An information processing apparatus100according to a second embodiment differs in the XYT image displayed at reverse playback. Specifically, since display of the XYT image suitable for forward playback and display of the XYT image suitable for reverse playback differ, an XYT image different from that for forward playback is displayed for reverse playback.

<Changing Viewpoint for Time Axis>

In displaying an image in which an address event signal is plotted in a coordinate space constituted by a first axis indicating the time axis and second axes related to space, the positions of the origin and the axes may be changed according to the method of display.FIGS.6A and6Billustrate images obtained by playing back the XYT image inFIG.4Ain the reverse direction.FIG.6Ais an XYT image viewed from the same viewpoint as inFIG.4A.FIG.6Bis an XYT image in which the origin of the coordinates is changed so that the direction of T-axis (time axis) is opposite to that ofFIG.6A. As is seen in the drawings, inFIGS.4A and6A, the past is displayed closer to the viewer, and the future farther from the viewer, while inFIG.6B, the past is displayed farther from the viewer, and the future closer to the viewer.

In reverse playback, an abnormality in the phenomenon that occurs in the future is often presumed retroactively. For this reason, not an XYT image in which the past is displayed on this side, as inFIG.6A, an XYT image in which the future is displayed on this side, as inFIG.6B, may be displayed for reverse playback. Accordingly, at reverse playback, the information processing apparatus100of the second embodiment displays an XYT image in which the viewpoint is changed to the opposite direction about the time axis from the direction at forward playback. Thus, for reverse playback, displaying an XYT image different from that for forward playback improves the visibility for the user.

<Changing Viewpoint for Space Axes>

By changing the viewpoint in the opposite direction from that for forward playback for the space axes (X-axis and Y-axis), in addition to the time axis, allows for display in which the past is displayed farther from the viewer, and the future closer to the viewer at reverse playback. However, the viewpoint may be reversed only for the time axis without changing the viewpoint for the space axes. This is because this makes it easy to associate the XYT image at forward playback with the XYT image at reverse playback, thereby allowing for comfort display in searching for the cause of the abnormality while repeating forward playback and reverse playback.

<Clipping from XYT Image

Another example is a use case in which part of the XYT image is to be clipped for display. For example, if the occurrence of a phenomenon different from those before and after a specific time is presumed from the XYT image, only the area around the specific time is to be extracted for display. There are two conceivable methods of display for this case. One is common moving-image display in which a frame format (XY image) that displays information on the distribution of changes in luminance in a two-dimensional coordinate space at some point in time is continuously updated for a specific period of time. Another one is a method of display in which a specific time is extracted from the XYT image and plots indicating information on changes in luminance are added to the XYT image space (hereinafter referred to as “plot-added XYT moving image”).

The information processing apparatus100of the second embodiment performs display suitable for each of forward playback and reverse playback as follows. The specific clipping time is from time T1(start time: first time) to time T2(end time: second time).

The image generation unit202generates a frame format image that displays information on the distribution of changes in luminance of the pixels integrated in a specific time range at some point in time. The display control unit203displays a frame format image at multiple points in time.FIG.7Aillustrates a frame format image generated from the forward playback image inFIG.6A.FIG.7Billustrates a frame format image generated from the reverse playback image inFIG.6B.FIGS.7A and7Bshow the directions in which the subject moves at forward playback and reverse playback with arrows. As are seen fromFIGS.7A and7B, the frame format image in which part of the reverse playback image is clipped displays the change in luminance of each pixel in reverse to that of the frame format image generated from the forward playback image.

In other words, the frame format image generated from the reverse playback image is displayed with the change in luminance of each pixel reversed, as the reverse XYT playback image is. Thus, the changes in luminance of each pixel displayed in the XYT image for looking down at the overall movement of the subject and in the frame format image in which detailed information of part of the XYT image match, thereby making it easy for the user to recognize the phenomenon occurring.

Plot images are generated in which pixels in which a change in luminance occurred at multiple points in time in a specific period are plotted in a coordinate space constituted by the first axis indicating the time axis and the second axes related to space. The generated plot images are displayed in a predetermined order by the display control unit203. In displaying the plot images in reverse chronological order, all the plot images generated in the specific period are displayed, and thereafter the plot images generated at individual points in time are deleted in the order from the second time to the first time. In displaying the plot images in chronological order, the plot images generated at individual points in time are superposed and displayed in the order from the first time to the second time.FIG.8Aillustrates plot-added XYT moving images generated from the forward playback image inFIG.4A.FIG.8Billustrates plot-added XYT moving images generated from the reverse playback image inFIG.6A.FIGS.8A and8Bschematically show the plotting in chronological order to express the moving images.

FIG.8Aillustrates moving images in which luminance change information is gradually added as plots onto an XYT space from time T1, and finally at time T2, all luminance changes generated in the specific time from time T1to time T2are displayed on the XYT image. In other words,FIG.8Aillustrates moving images in which future luminance changes are gradually superposed on the past luminance changes. The use of such plot-added XYT moving images allows the user to easily grasp the causal relationship between a phenomenon generated at the time of interest and a phenomenon before it.

In contrast,FIG.8Billustrates moving images starting from a state in which all the luminance change information generated from time T2to time T1are displayed on the XYT image, from which the plots of future luminance change information are gradually deleted in reverse chronological order. In other words,FIG.8Billustrates moving images in which future changes in luminance are gradually deleted from the image in which all the luminance changes generated in the specific time are superposed. The use of plot-added XYT moving images beneficially makes it easy to, at reverse playback, grasp the causal relationship between a phenomenon occurring at the time of interest and a phenomenon that has occurred before that.

In other words, at forward playback, future luminance change information is gradually added to the past luminance change information, and at reverse playback, the future luminance change information is gradually deleted, with the past luminance change information left. This configuration beneficially makes it easy to grasp the causal relationship between a phenomenon generated at the time of interest and a phenomenon before it.

<Viewpoint in Future Deleted Display>

As shown inFIG.8B, for the moving images in which luminance changes that occur after the present time are gradually deleted, the past may be displayed farther from the viewer, and the future may be displayed closer to the viewer. In contrast, for the moving images in which past luminance changes are gradually added to future luminance changes, the past may be displayed closer to the viewer, and the future may be displayed farther to the viewer, as shown inFIG.8C.

In other words, the viewpoint may be changed to the opposite side about the time axis at reverse playback between a case in which future luminance change information is gradually deleted for display, with past luminance change information left, and a case in which past luminance changes are gradually added to the future luminance changes.

FIG.8Bshows an example in which future luminance changes are gradually deleted to facilitate grasping the causal relationship between a phenomenon that occurred at the time of interest and a phenomenon that occurred before that. However, there may be a use case in which the cause of a phenomenon that occurred after the time of interest is to be ascertained. In this case, as shown inFIG.8C, moving images in which luminance change information is gradually added from time T2as plots onto the XYT space, and finally at time T1, all the luminance changes that occurred in the specific time from time T2to time T1are displayed on the XYT image may be displayed. In other words, moving images in which past luminance changes are gradually added to future luminance changes may be displayed.

Other Embodiments

This application claims the benefit of Japanese Patent Application No. 2020-186564 filed Nov. 9, 2020, which is hereby incorporated by reference herein in its entirety.