Information processing apparatus, information processing system, information processing method, and storage medium

The information processing apparatus includes an acquisition unit configured to acquire a plurality of videos captured by a plurality of imaging apparatuses, an extraction unit configured to extract one or more pieces of object information each representing a trajectory of an object from each of the plurality of videos acquired by the acquisition unit, and a generation unit configured to generate a synopsis video obtained by gathering objects on one background image based on the object information extracted by the extraction unit.

BACKGROUND OF THE INVENTION

Field of the Invention

The present disclosure relates to an information processing apparatus, an information processing system, an information processing method, and a storage medium.

Description of the Related Art

In recent years, utilization of a video having a long recording time has increased. A video captured by a network camera installed in a store or on a road, for example, is used in various kinds of applications such as remote monitoring, crime prevention, and marketing for analyzation of a customer traffic state in a store. The video is recorded in a recording device such as a digital video recorder (DVR) and a network video recorder (NVR) so that the video can be used later. A video analysis system that can count objects such as a person and a vehicle included in the video and can detect an object passing a line drawn in the video by a user has been known.

However, the video having the long recording time often includes a time period in which there is no movement or no change for a long time in the video. As a result, checking of the recorded image requires a long time, which may cause issues such as overlooking of a target.

Japanese Patent No. 4972095 discusses a technique in which moving targets displayed at different times in an original video are displayed at a time to summarize the original video in a short-time video. The technique enables checking of only moving objects in the recorded video in a short time, which makes it possible to reduce possibility of overlooking the target.

SUMMARY OF THE INVENTION

According to an aspect of the present disclosure, an information processing apparatus changes an appearance order of targets detected in a video to generate a synopsis video, and includes an acquisition unit configured to acquire a plurality of videos captured by a plurality of imaging apparatuses, an extraction unit configured to extract one or more pieces of object information each representing a trajectory of an object from each of the plurality of videos acquired by the acquisition unit, and a generation unit configured to generate a synopsis video obtained by gathering objects on one background image based on the object information extracted by the extraction unit.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present disclosure are described in detail below with reference to accompanying drawings. The exemplary embodiments described below are merely examples of implementation of the present disclosure, and are appropriately corrected or modified depending on a configuration of an apparatus to which the present disclosure is applied and various kinds of conditions. The present disclosure is not limited to the exemplary embodiments described below. Further, not all combinations of features described in the exemplary embodiments are necessarily essential to the solving means of the present disclosure.

FIG. 1is a system configuration diagram of a synopsis video generation system1according to a first exemplary embodiment of the present disclosure. As illustrated inFIG. 1, the synopsis video generation system1has a configuration in which imaging apparatuses101to103, a terminal105, and a storage106are connected through a network (internet protocol (IP) network)104. The imaging apparatuses101to103are, for example, network cameras. The terminal105is an apparatus that processes predetermined information. Thus, the terminal105may be referred to as an information processing apparatus. Each of the imaging apparatuses101to103is installed on a ceiling of a hotel or a train and continuously performs imaging to capture a long-time video (including audio data). Each of the imaging apparatuses101to103transmits an original video that is acquired by imaging and is to be subjected to analysis processing to the terminal105via the network104.

The terminal105is an apparatus that displays original videos and various kinds of information from the imaging apparatuses101to103and remotely operates the imaging apparatuses101to103. The terminal105is, for example, a personal computer. The personal computer is an example of a general-purpose computer and is referred to as a PC, in the description below. The terminal105has functions of receiving the original videos from the imaging apparatuses101to103via the network104and displaying the original videos. The terminal105further includes a function of issuing various kinds of commands to the imaging apparatuses101to103.

The storage106is a storage device that stores the original videos captured by the imaging apparatuses101to103and various kinds of setting values. The storage106may be a recording device such as a digital video recorder (DVR) and a network video recorder (NVR). The storage106may store an image acquired from an external device (not illustrated) in addition to the videos from the imaging apparatuses101to103, The external device includes a portable storage medium such as a secure digital (SD) memory card and a digital versatile disc (DVD) in addition to another terminal.

While three imaging apparatuses101to103, one terminal105, and one storage106are illustrated inFIG. 1, the number of devices is not particularly limited thereto. In addition, the terminal105and the storage106may be connected not via the network104but by a physical connection such as a universal serial bus (USB) connection. Furthermore, the storage106may be incorporated in the terminal105.

FIG. 2is a conceptual diagram illustrating a usage example of the synopsis video generation system1. In the example ofFIG. 2, the imaging apparatuses101to103are respectively installed in corridors of first to third floors of a hotel201. Original videos202to204illustrated inFIG. 2are respectively captured by the imaging apparatuses101to103. As understood fromFIG. 2, backgrounds in the original videos202to204are substantially the same although illumination and decoration are slightly different.

WhileFIG. 2illustrates the example of the hotel, a scene in which the backgrounds in the original videos captured by the respective imaging apparatuses are similar to one another is not limited to the hotel. For example, similar original videos are captured by a plurality of imaging apparatuses installed in cars of a train.

FIG. 3is a diagram illustrating an example of a hardware configuration of the terminal105. As illustrated inFIG. 3, the terminal105includes a central processing unit (CPU)301, an input unit302, a display unit303, a memory304, a digital interface unit (I/F)305, and a communication terminal306.

The CPU301integrally controls the components of the terminal105. The CPU301executes a program stored in the memory304. The CPU301may perform control by using hardware.

The memory304is a storage device used as a storage area of the program to be executed by the CPU301, a work area during execution of the program, and a data storage area.

The input unit302is a functional unit that receives an input operation performed by a user, and may include various kinds of input devices such as a button, a cross key, a touch panel, and a mouse.

The input operation received by the input unit302includes an input operation of an instruction to transmit the various kinds of commands to the imaging apparatuses101to103. Upon receiving the input operation, the input unit302notifies the CPU301of reception of the input operation. The CPU301generates commands to the imaging apparatuses101to103in response to the notification. Then, the CPU301instructs the I/F305to transmit the generated commands to the imaging apparatuses101to103.

The input unit302also performs processing to receive input of a user response to an inquiry message to the user, which is generated by the CPU301executing the program stored in the memory304.

The display unit303is a functional unit that displays a result of the processing performed by the CPU301. More specifically, the display unit303may be a display device such as a liquid crystal display device, a plasma display device, or a cathode-ray tube (CRT) display device. In the case where the CPU301generates the inquiry message to the user, the display unit303transmits the message to the user by displaying the message.

The I/F305is a functional unit that receives an instruction from the CPU301and transmits the commands to the imaging apparatuses101to103via the communication terminal306. The I/F305also performs processing to receive responses to the commands and the streamed original videos from the imaging apparatuses101to103via the communication terminal306. The original videos received by the I/F305are supplied to the CPU301. The CPU301decodes and expands the original videos supplied from the I/F305and supplies the original videos to the display unit303. The display unit303displays the original videos supplied from the CPU301.

The communication terminal306includes a local area network (LAN) terminal to which a LAN cable is connected. While the hardware configuration of the terminal105according to the present exemplary embodiment has been described above, the hardware configuration illustrated inFIG. 3illustrates a suitable example of the terminal105according to the present exemplary embodiment, and the hardware configuration is not limited to the configuration inFIG. 3. For example, various modifications and alternations, such as provision of an audio input unit and an audio output unit, can be made within the scope of the present disclosure.

FIG. 4is a functional block diagram illustrating functions of the terminal105implemented by the CPU301illustrated inFIG. 3. Hereinafter, the functions of the terminal105for generating a synopsis video according to the present exemplary embodiment are described in detail with reference toFIG. 4.FIG. 4mainly illustrates functional blocks relating to generation of the synopsis video, and other functions of the terminal105are not limited to the illustrated configuration.

As illustrated inFIG. 4, the terminal105includes a video processing unit401, a control unit410, and a calculation unit411as functional units. The video processing unit401is a functional unit that performs processing to generate the synopsis video from an input video. The video processing unit401includes a video acquisition unit402, a background image extraction unit403, an object extraction unit404, an additional data generation unit405, a first storage unit406, an arrangement processing unit407, a synopsis generation unit408, and a second storage unit409.

The control unit410and the calculation unit411are functional units that process commands from program data stored in the memory304(seeFIG. 3), which is included inside the terminal105, and perform calculation processing. More specifically, the control unit410performs control to, for example, transmit a command included in the program data stored in an external memory of the CPU301to the video processing unit401, and transmit an instruction from the video processing unit401to the outside of the CPU301. The calculation unit411performs various kinds of calculation processing and processing to compare images based on the command provided by the control unit410. As a result, on the terminal105, the user can control the imaging apparatuses101to103and provide various instructions on a screen while viewing the display unit303. The control unit410also performs processing to reproduce the synopsis video generated by the video processing unit401.

The video acquisition unit402acquires the original videos from the imaging apparatuses101to103and performs processing to determine a moving image format of the original videos so that each of frames constituting the original videos can be analyzed. The video acquisition unit402also performs conversion of the moving image format as necessary.

The background image extraction unit403performs processing to extract a background image from each of the plurality of original videos acquired by the video acquisition unit402. For example, a scene including no object may be previously captured by each of the imaging apparatuses101to103, and videos obtained as a result may be stored in the first storage unit406in association with the respective imaging apparatuses101to103. Then, an area where no object is detected may be extracted and may be partially updated.

FIG. 5is a diagram illustrating an example of background images extracted by the background image extraction unit403. Background images601to603illustrated inFIG. 5are respectively extracted from the original videos202to204illustrated inFIG. 2.

The background image extraction unit403further superimposes the plurality of background images acquired from the respective imaging apparatuses101to103, and determines similarity among them. In a case where the similarity is greater than or equal to a predetermined threshold as a result of the determination, the background image extraction unit403determines that the original videos from the imaging apparatuses101to103can be summarized on the same background image. Then, the background image extraction unit403performs processing to select a background image to be used in the synopsis video from among the plurality of extracted background images and to store the selected background image in the first storage unit406. The detail of the processing is described below. In a case where the similarity is less than the predetermined threshold, the background image extraction unit403determines that the original videos from the imaging apparatuses101to103cannot be summarized on the same background image. Then, the background image extraction unit403stores the plurality of extracted background images in the first storage unit406.

The object extraction unit404performs processing to extract one or more pieces of object information each representing a trajectory of an object from each of the plurality of original videos acquired by the video acquisition unit402and to store the object information in the first storage unit406. For example, the object is extracted by using a known method such as a background difference method and a human body detection method. The object extraction unit404uses a center of an extracted pixel area as coordinates of the object and uses the pixel area as a range of the object. The object information specifically includes time-series information on the coordinates (x, y) and the range of the extracted object. The object extraction unit404also stores information indicating in which frame the detected object appears and in which frame the detected object disappears (starting time and duration) in the first storage unit406.

The extraction by the object extraction unit404may be performed by using the technique discussed in Japanese Patent No. 4972095. In addition, any extraction method may be used as long as the object can be extracted.

The additional data generation unit405adds, as one of metadata, identification information on an imaging apparatus to the object information in order to grasp by which imaging apparatus the object extracted by the object extraction unit404has been acquired. As the identification information on the imaging apparatus, an IP address, an identification number, and a camera name of the imaging apparatus are expected. A color of a time stamp or a detection frame described below that is displayed to be superimposed on the object is determined based on the identification information on the imaging apparatus. However, any information can be used as the identification information on the imaging apparatus as long as the information can associate the imaging apparatus capturing the object with the object in the synopsis video.

The first storage unit406stores one or the plurality of background images stored by the background image extraction unit403and the one or more pieces of object information stored by the object extraction unit404. As a result, the first storage unit406functions as a storage area of the information necessary for implementing the functions described below.

The arrangement processing unit407performs processing to once arrange all of the one or more pieces of object information extracted by the object extraction unit404in a time-series order. The reason for performing the processing is because, if the one or more pieces of object information extracted from the imaging apparatuses are superimposed on one video, the pieces of object information may be displayed at the same coordinates in an overlapping manner at the same timing. To grasp occurrence/non-occurrence of such a situation, the arrangement processing unit407uses the technique discussed in Japanese Patent No. 4972095 to collectively arrange all the pieces of the object information in a time-series order. Then, to prevent the pieces of object information from being displayed in an overlapping manner, the arrangement processing unit407performs processing to shift the starting time of each of the pieces of object information and to rearrange the object information. The detail of the rearrangement processing is described below.

The first storage unit406may store the pieces of object information in a table collectively including coordinates information, time information (staying time), and imaging apparatus information.

The synopsis generation unit408performs processing to summarize the original videos based on arrangement generated by the arrangement processing unit407(i.e., one or more pieces of object information processed by the arrangement processing unit407) and the background images stored in the first storage unit406by the background image extraction unit403. The detail thereof will be described below. A synopsis video obtained by gathering the one or more pieces of object information extracted from the imaging apparatuses on one background image is generated by summarization processing. The synopsis generation unit408also performs processing to add information (e.g., color information described above) for identifying the imaging apparatus corresponding to each piece of the Object information disposed in the synopsis video.

The synopsis generation unit408once stores the generated synopsis video in the second storage unit409. As a result, the synopsis video stored in the second storage unit409is put into a state of waiting for rasterization in a display area, and is reproduced by the control unit410in response to an instruction by the user.

Next, a flow of the processing to generate one synopsis video based on the plurality of original videos from the plurality of imaging apparatuses is specifically described with reference to a flowchart illustrated inFIG. 6as well asFIG. 4. The processing in the flowchart is performed by the CPU301based on a control program read out to the memory304. In the present exemplary embodiment, to generate the synopsis video, chronological order of appearance of the targets is changed from the order in the original videos without moving spatial positions of the pieces of the object information from the plurality of imaging apparatuses.

The arrangement processing unit407determines the starting time of a motion trajectory of each of the objects extracted from the original videos captured by the plurality of imaging apparatuses in the synopsis video so as to minimize the number of collisions among motion trajectories of all of the objects and to reduce a reproduction time.

Collision among the motion trajectories of the objects is described. The arrangement processing unit407calculates a collision cost Colij(k) between targets i and j by using the following formula (1). In the formula (1), k is a time difference in the starting time between the targets i and j, xtiand ytiare an x coordinate and a y coordinate of a center of the target i at time t, and is a radius of the target i at the time t. In addition, Tij(k) is a period in which the motion trajectories of both of the targets i and j appear in the video when the time difference in the starting time is k. Further, an expression inside an absolute value symbol on a right side in the formula (1) has a relatively large value representing presence of collision in a case where a distance between the center of the target i and a center of the target j is less than a sum of the radius of the target i and a radius of the target j and has a relatively small value representing absence of collision in other cases. Accordingly, the larger the collision cost Colij(k) is, the larger the number of frames in which the targets i and j collide with each other is.

The arrangement processing unit407uses a simulated annealing method to determine the starting times at which the collision cost Colij(k) becomes the minimum in all of combinations of the targets. By using the simulated annealing method, it is possible to efficiently calculate a nonlinear programming problem with a limit in range.

Accordingly, the arrangement processing unit407uses the simulated annealing method to determine the starting time of each of the targets, thereby determining an optimum starting time of each of the targets in the synopsis video. The arrangement processing unit407generate the synopsis video in the above-described manner by changing an order of appearance while maintaining the spatial positions of the targets in the original videos. More specifically, the synopsis video is generated in such a manner that images of the targets each extracted using a pixel mask from the original videos are pasted on the background image based on the calculated starting times.

FIG. 6is a flowchart illustrating the processing performed by the terminal105according to the present exemplary embodiment. As illustrated inFIG. 6, in step S101, the video acquisition unit402of the terminal105first acquires the original videos from the respective imaging apparatuses. In this example, the original videos are acquired from the respective imaging apparatuses; however, videos stored in an external memory or a storage may be acquired as the original videos.

Next, in step S102, the object extraction unit404of the terminal105extracts the above-described object information from each of the acquired original videos, and stores the object information in the first storage unit406.

Next, in step S103, the background image extraction unit403of the terminal105extracts the background image from each of the original videos captured by the respective imaging apparatuses, and selects the background image to be used in the synopsis video from among the plurality of extracted background images.

For example, the background image is selected by using one of two selection methods, i.e., (i) to select a background image of the imaging apparatus with the largest number of pieces of extracted object information, and (ii) to select the background image of the imaging apparatus with the smallest number of pieces of extracted object information. In the case of using the former selection method, it is possible to minimize influence in a case where the backgrounds are largely different among the imaging apparatuses. In the case of using the latter selection method, it is possible to enhance accuracy of the background image because an update frequency of the background image is increased based on appearance of the object.

While a still image is basically used as the background image, the background image extraction unit403may select a different still image corresponding to a different time of day at which each of the original videos is captured. This makes it possible to appropriately deal with change in ambient light between daytime and nighttime.

Next, in step S104, the additional data generation unit405of the terminal105performs processing to add the identification information (e.g., color, number, character) on the imaging apparatus to an outer rim (detection frame) of the object information or the time stamp (time information) displayed to be superimposed on the object information displayed in the synopsis video. The processing is performed to enable the user viewing the synopsis video to grasp the imaging apparatus that has captured each of the objects.

FIGS. 7A to 7Care diagrams each illustrating an example of the synopsis video generated as a result of the processing in step S104.FIGS. 7A to 7Ceach illustrate a case where the background image603illustrated inFIG. 5is selected as the background image.

FIG. 7Aillustrates an example in which colors are added to the detection frames. However, since chromatic colors are unusable in patent drawings, types of lines are changed to represent different colors as an alternative illustration inFIG. 7A. More specifically, a detection frame of an object702captured by the imaging apparatus101on the first floor (seeFIG. 2) is indicated by a dashed line. A detection frame of an object703captured by the imaging apparatus102on the second floor is indicated by a long dashed line. A detection frame of an object704captured by the imaging apparatus103on the third floor is indicated by a solid line.

FIG. 7Billustrates an example in which a color is added to the time stamp. In the example, a time stamp “12:30” of the object702captured by the imaging apparatus101on the first floor is indicated by red characters. A time stamp “10:03” of the object703captured by the imaging apparatus102on the second floor is indicated by green characters. A time stamp “20:15” of the object704captured by the imaging apparatus103on the third floor is indicated by yellow characters.

FIG. 7Cillustrates an example in which character information is added to the time stamp. In the example, characters “_1F” are added to the time stamp “12:30” of the object702captured by the imaging apparatus101on the first floor. Characters “_2F” are added to the time stamp “10:03” of the object703captured by the imaging apparatus102on the second floor, Characters “_3F” are added to the time stamp “20:15” of the object704captured by the imaging apparatus103on the third floor.

Next, in step S105, the arrangement processing unit407of the terminal105once arranges the one or more pieces of object information extracted by the object extraction unit404in a time-series order. More specifically, the arrangement processing unit407arranges all the pieces of the object information stored in the first storage unit406and analyses the object information.

In step S106, the arrangement processing unit407of the terminal105detects an overlapping area where a plurality of pieces of object information is displayed at the same coordinates in an overlapping manner at the same timing in the synopsis video from the object information arranged in step S105. The reason for performing the processing is because, when the objects captured by the respective imaging apparatuses are displayed in one synopsis video, the objects may be present at the same coordinates at the same timing.

FIG. 8is a diagram illustrating the overlapping area. The original videos202to204respectively captured by the imaging apparatuses101to103are illustrated on an upper side ofFIG. 8. As illustrated inFIG. 8, the original videos202to204include objects801to807. Further, in each diagram on the upper side ofFIG. 8, a trajectory corresponding to each of the objects801to807is illustrated behind each of the objects801to807. InFIG. 8, the trajectories of the objects801to803captured by the imaging apparatus101are indicated by long dashed lines, the trajectories of the objects804and805captured by the imaging apparatus102are indicated by dashed lines, and the trajectories of the objects806and807captured by the imaging apparatus103are indicated by solid lines.

An all object information arrangement diagram808that has a lateral axis representing a coordinate axis x and a vertical axis representing a time axis t is illustrated on a lower side ofFIG. 8. The trajectories illustrated in the diagrams on the upper side ofFIG. 8are illustrated in the all object information arrangement diagram808. Among the trajectories illustrated in the all object info nation arrangement diagram808, a trajectory809represents the trajectory of the object804, and a trajectory810represents the trajectory of the object802. It is understood fromFIG. 8that the two trajectories809and810are overlapped in the same time period.

In a case where the overlapping area is detected, in step S107, the arrangement processing unit407performs processing to shift the starting time of at least one piece of the object information so as to eliminate the overlapping area and to rearrange the object information. The processing is performed in the above-described manner.

FIGS. 9A and 9Bare diagrams each illustrating an example of the processing in step S107. The reference numerals808to810inFIGS. 9A and 9Brepresent the same components illustrated in the diagram on the lower side ofFIG. 8.

FIG. 9Aillustrates an example in which two pieces of object information to be displayed in an overlapping manner are displayed within the shortest time. In this case, the arrangement processing unit407corrects the starting time of one of the two pieces of object information to be displayed in an overlapping manner based on a time immediately after the end of the other piece of object information. In the example ofFIG. 9A, a correction is made so that the starting time of the trajectory810is a time immediately after the end of the trajectory809. An order of display of the object information (display order in the synopsis video) in this case preferably is an ascending order of management numbers of the imaging apparatuses.

FIG. 9Billustrates an example in which the plurality of pieces of object information captured by the same imaging apparatus is displayed so as to be temporally overlapped with one another. In this case, the arrangement processing unit407corrects the starting time of at least one of the two pieces of overlapping object information so that, for example, the plurality of pieces of object information from the imaging apparatus designated by the user is displayed at the same timing as much as possible. In the example ofFIG. 9B, the starting time of the trajectory809is corrected so that the trajectory809and a trajectory of the other object captured by the imaging apparatus102are displayed at the same timing.

Alternatively, in place of shifting the starting time as described above, one of the two pieces of overlapping object information may be pop-up displayed or thumbnail-displayed. More specifically, the arrangement processing unit407may add predetermined flag information to one of the two pieces of object information to be displayed in an overlapping manner. The object information added with the flag information may be pop-up displayed or thumbnail-displayed when the control unit410reproduces the synopsis video. As for the object information that is not pop-up displayed or thumbnail-displayed but is normally displayed on the background image, the object information obtained from the imaging apparatus capturing the background image in use is preferably selected.

Whether to gather the object information from the plurality of imaging apparatuses in one synopsis video may be determined based on density of the objects in the synopsis video. For example, in a case where the object information from the plurality of imaging apparatuses is collectively displayed in one synopsis video, an area occupied by the objects may exceed a predetermined ratio (e.g., 60%) of the screen. In such a case, in place of gathering the object information from the plurality of imaging apparatuses in one synopsis video, a synopsis video for each of the imaging apparatuses may be generated.

Next, in step S108, the synopsis generation unit408generates a synopsis video that is obtained by gathering the one or more pieces of object information extracted in step S102on one background image based on the arrangement generated by the arrangement processing unit407(through the processing in step S107).

FIG. 10is a diagram illustrating an all object information arrangement diagram1001in the synopsis video. The all object information arrangement diagram1001is generated based on the all object information arrangement diagram808illustrated inFIG. 9AorFIG. 9B.

It is understood fromFIG. 10that the summarization processing is processing to change the starting time of each piece of the object information within a range not overlapping with another piece of the object information. This makes it possible to reduce a reproduction time of the video as much as possible within a range where the pieces of object information do not overlap with one another. In a case where a larger effect can be expected from the summarization processing if the object information is moved until the object information partially overlaps with another piece of the object information, processing to thin the overlapping portion (deletion of frames) may be performed.

The synopsis generation unit408may collectively arrange the object information for each corresponding imaging apparatus in the synopsis video.

Finally, in step S109, the control unit410of the terminal105reproduces the synopsis video. Thus, the series of processing to generate the synopsis video from the original videos captured by the plurality of imaging apparatuses ends.

As described above, in the synopsis video generation system1according to the present exemplary embodiment, the plurality of original videos captured by the plurality of imaging apparatuses is collectively reproduced as one synopsis video. This makes it possible to check the videos in a short time as compared with the existing technique.

A configuration of an imaging apparatus according to a second exemplary embodiment is described. Description of parts equivalent to the parts illustrated in the first exemplary embodiment is omitted, in the above-described first exemplary embodiment, the configuration to gather the plurality of original videos from the plurality of imaging apparatuses in one synopsis video is described. In the second exemplary embodiment, processing when an object is selected during reproduction of the synopsis video is described.

FIG. 11is a flowchart illustrating processing performed by the terminal105according to the present exemplary embodiment. Hereinafter, a specific operation of the terminal105when the user selects an object during reproduction of the synopsis video that is generated based on the plurality of original videos from the plurality of imaging apparatuses is described with reference toFIG. 11. Processing in steps S101to S109inFIG. 11is similar to the processing in steps S101to S109illustrated inFIG. 5. Thus, description thereof is omitted.

During reproduction of the synopsis video in step S109, in step S201, the control unit410receives an instruction operation to select the object information from the user. The selection instruction operation may be performed, with a mouse, by the user viewing a monitor displaying the synopsis video. In a case where the selection instruction operation is performed by the user (YES in step S201), in step S202, the control unit410switches the video being reproduced to the synopsis video that is generated from only the original video captured by the imaging apparatus that has captured the object corresponding to the selected object information. On the other hand, in a case where reproduction of the synopsis video ends without the selection instruction operation (NO in step S201), the control unit410ends the processing without performing the processing in step S202.

A diagram on a left side inFIG. 12is a diagram illustrating a synopsis video205in which the original videos202to204illustrated inFIG. 8are collectively displayed. A diagram on a right side inFIG. 12is a diagram illustrating a synopsis video206from only the original video202illustrated inFIG. 8. If the user selects the object803(surrounded by a black frame) in the synopsis video205in the diagram on the left side inFIG. 12, the terminal105summarizes the original video202captured by the imaging apparatus101, which has captured the object803, and reproduces the synopsis video as illustrated in the diagram on the right side inFIG. 12.

A diagram on a left side inFIG. 13is a diagram illustrating the all object information arrangement diagram1001same as in the diagram inFIG. 10. A diagram on a right side inFIG. 13is a diagram illustrating an all object information arrangement diagram1304of the synopsis video206illustrated in the diagram on the right side inFIG. 12. Trajectories1301and1302illustrated inFIG. 13are trajectories of the objects803and801, respectively. As understood fromFIG. 13, in the case where only the original video202captured by the imaging apparatus101is summarized and the synopsis video is reproduced, reproduction may end in a time shorter than the reproduction time of the synopsis video205in which the original videos202to204are collectively displayed (the all object information arrangement diagram1304is smaller than the all object information arrangement diagram1001).

While not illustrated, in a case where the object the same as the object of the selected object information is detected by the plurality of imaging apparatuses, a synopsis video obtained by gathering the plurality of pieces of the object information on the object may be reproduced. Alternatively, the synopsis videos obtained from only the respective original videos captured by the plurality of imaging apparatuses that has captured the object may be sequentially switched and reproduced.

As described above, in the synopsis video generation system1according to the present exemplary embodiment, if the user finds an object to be specifically checked while checking the synopsis video obtained by gathering the object information included in the plurality of original videos, the user can check the object information in detail.

Even when the object information is not selected, the synopsis video obtained by gathering the object information included in the plurality of original videos and the synopsis videos from only the respective original videos captured by the respective imaging apparatuses may be switched.

While the exemplary embodiments of the present disclosure have been described in detail with reference to the drawings, the present disclosure is not limited to these specific exemplary embodiments and includes various modes without departing from the scope of the present disclosure. Parts of the above-described exemplary embodiments may be appropriately combined.

For example, in a synopsis video of videos captured inside a moving object such as a train, information on an imaging position may be acquired using a positional information system such as a global positioning system (GPS). The acquired information on the imaging position may be added as one of the above-described metadata to the object information. This makes it possible to add positional information representing that the object has been captured at which station to each of the objects in the synopsis video by using colors, characters, and the like.

The present disclosure can be realized by supplying a program implementing a part or one or more of the functions of the above-described exemplary embodiments to a system or an apparatus via a network or a storage medium, and causing one or more processors in a computer (or a CPU, microprocessor unit (MPU), etc.) of the system or the apparatus to read out and execute the program. Further, the program may be provided by being recorded in a computer-readable recording medium.

Further, the present disclosure is not limited to the configuration in which the functions of the exemplary embodiments are implemented by execution of the program read out by the computer. For example, an operating system (OS) or the like operating on the computer may perform a part or all of actual processing based on an instruction from the program, and the functions of the above-described exemplary embodiment may be implemented by the processing.

OTHER EMBODIMENTS

This application claims the benefit of Japanese Patent Application No. 2019-188340, filed Oct. 15, 2019, which is hereby incorporated by reference herein in its entirety.