Source: https://patents.google.com/patent/JP2007074503A/en
Timestamp: 2020-02-16 19:16:19
Document Index: 636237330

Matched Legal Cases: ['art 5', 'art 3', 'art 4', 'art 5', 'art 6', 'art 7']

JP2007074503A - Dynamic image editing device - Google Patents
Dynamic image editing device Download PDF
JP2007074503A
JP2007074503A JP2005260625A JP2005260625A JP2007074503A JP 2007074503 A JP2007074503 A JP 2007074503A JP 2005260625 A JP2005260625 A JP 2005260625A JP 2005260625 A JP2005260625 A JP 2005260625A JP 2007074503 A JP2007074503 A JP 2007074503A
JP2005260625A
JP4513699B2 (en
航一 木下
由紀子 柳川
修一郎 築地
2005-09-08 Application filed by Omron Corp, オムロン株式会社 filed Critical Omron Corp
2005-09-08 Priority to JP2005260625A priority Critical patent/JP4513699B2/en
2007-03-22 Publication of JP2007074503A publication Critical patent/JP2007074503A/en
2010-07-28 Publication of JP4513699B2 publication Critical patent/JP4513699B2/en
<P>PROBLEM TO BE SOLVED: To generate a dynamic image which maintains visibility of a noticed object and which does not lose its uplifting feeling. <P>SOLUTION: A position of an object is acquired in each frame of an inputted dynamic image, a position of a cut frame is determined within a frame based on the position of the object so as to put the object within the cut frame, and the dynamic image is reconstructed by cutting an image enclosed by the cut frame. At this time, as for components in a vertical direction of the cut position, the position of the cut frame is determined so that the influence of the position of the object may become smaller compared with components in a horizontal direction. <P>COPYRIGHT: (C)2007,JPO&INPIT
The present invention relates to a technique effectively applied to an apparatus for framing a moving image.
Due to the recent miniaturization of image pickup devices, camera shake during image pickup has become prominent in image pickup results, which is a problem. For amateurs, it is difficult to accurately frame the subject. Due to such blurring and inaccurate framing, there arises a problem that the visibility of the subject of interest is lowered.
In order to solve such a problem, an automatic framing technique has been proposed in which the subject is held at the center of the screen during shooting. For example, a technique has been proposed in which a wide-area image is captured and a tracking image in which the subject does not go out of the frame is obtained by continuously cutting out partial images including the subject of interest (Patent Document 1). reference).
In addition, a technique for acquiring and displaying an image in which a person is placed at the center of the screen (see Patent Document 2) and a technique for capturing an image of a manually designated subject (see Patent Document 3) have been proposed. .
JP 2004-207884 A JP 2000-078564 A Japanese Patent Laid-Open No. 06-038214
In any of the prior arts, an object of interest is to improve the visibility of an object by extracting a subject of interest from a captured image and performing framing by capturing the subject of interest at the center of the screen. In a moving image framed in this manner, the subject of interest always exists at the center of the screen, and thus visibility is improved. However, in the video by such a method, there is a problem that the position of the subject of interest does not move from the center of the screen and the feeling of dynamism is lost. Such a problem does not occur in imaging with a surveillance camera whose main purpose is to improve the visibility of the subject of interest. However, there is a problem when the purpose is to capture an everyday scene, such as imaging with a home digital video camera.
Accordingly, an object of the present invention is to solve these problems and to provide an apparatus and a method for generating a moving image that does not lose its dynamic feeling while maintaining the visibility of the subject of interest.
In order to solve the above problems, the present invention has the following configuration. The present invention is a moving image editing apparatus, and includes a position acquisition unit and a cutout unit. The position acquisition unit acquires the position of the object in each frame of the input moving image. The cutout means determines the position of the cutout frame in the frame based on the reference position preset in the frame and the position of the target so that the target is contained in the cutout frame, and is surrounded by the cutout frame. Reconstruct moving images by cutting out images. At this time, the cutout unit determines the position of the cutout frame so that the influence of the position of the target object on the vertical component of the cutout position is smaller than the horizontal component. The target object may be arbitrarily set, for example, a human head, face, car, ball or the like.
According to the present invention configured as described above, when a part of each frame of the moving image is cut out and a new moving image is reconstructed, the position of the cutting frame is determined by the cutting means. For this reason, since the component in the horizontal direction changes according to the position of the object, the visibility of the subject of interest (object) can be maintained. Furthermore, the vertical component is less affected by the position of the object than the horizontal component, in other words, the followability to the position of the object is low. For this reason, the vertical position of the object within the clipping frame varies greatly from frame to frame in each frame. For this reason, since the vertical position change is maintained in the cut out image, it is possible to generate a moving image that does not lose its dynamic feeling.
The cut-out means of the present invention obtains the difference between the reference position and the position of the object, multiplies the horizontal component of the difference by the first value, and smaller than the first value for the vertical component of the difference. The position of the cutout frame may be determined by multiplying the second value and using the obtained both values and the reference position.
The clipping means of the present invention obtains a difference between the reference position and the position of the object in a plurality of frames, a time-series horizontal component in consideration of a time-series change between frames for the horizontal component and the vertical component of the difference, and Calculate the time-series vertical component, multiply the time-series horizontal component by the first value, multiply the time-series vertical component by the second value smaller than the first value, and use both the obtained values and the reference position. The position of the cutout frame may be determined.
The clipping means of the present invention may be configured to calculate a time-series horizontal component and a time-series vertical component by performing Fourier transform.
The present invention may be realized by a program being executed by an information processing apparatus. That is, the present invention can specify the processing executed by each of the above-described means as a program for causing the information processing apparatus to execute, or a recording medium on which the program is recorded. Further, the present invention may be specified by a method in which the information processing apparatus executes the processing executed by each of the above-described means. When the present invention is specified by the method, it does not matter whether the information processing apparatus performs only image editing or plays back (screens) while editing the image. Furthermore, the present invention may be specified by a recording medium on which a moving image edited by the above method is recorded.
According to the present invention, it is possible to generate a moving image that maintains the visibility of the subject of interest and does not lose its dynamic feeling.
[System Configuration] FIG. 1 is a diagram illustrating an example of functional blocks of the moving image editing apparatus 1. Hereinafter, the moving image editing apparatus 1 will be described as an example of the moving image editing apparatus. The moving image editing apparatus 1 detects / tracks a person's head as an object and uses it as a framing target. However, the moving image editing apparatus 1 detects / tracks another object (such as a car or a ball) and uses it as a framing target. It may be configured.
The moving image editing apparatus 1 includes a CPU (Central Processing Unit), a main storage device (RAM), an auxiliary storage device, and the like connected via a bus in terms of hardware. The moving image editing apparatus 1 is loaded with various programs (OS, applications, etc.) stored in the auxiliary storage device and executed by the CPU, whereby the image input unit 2, the temporary storage unit 3, the head It functions as an apparatus including the detection unit 4, the head tracking unit 5, the cutout unit 6, and the like. The head detection unit 4, the head tracking unit 5, and the cutout unit 6 are realized by executing a program by the CPU. Moreover, the head detection unit 4, the head tracking unit 5, and the cutout unit 6 may be configured as dedicated chips. Next, each functional unit included in the moving image editing apparatus 1 will be described.
<Image Input Unit> The image input unit 2 functions as an interface for inputting moving image data to the moving image editing apparatus 1. The image input unit 2 inputs moving image data to the moving image editing apparatus 1. The image input unit 2 may be configured using any existing technique for inputting moving image data to the moving image editing apparatus 1.
For example, moving image data may be input to the moving image editing apparatus 1 via a network (for example, a local area network or the Internet). In this case, the image input unit 2 is configured using a network interface. In addition, moving image data may be input to the moving image editing apparatus 1 from a digital video camera, a personal computer, a recording device (for example, a hard disk drive), or the like. In this case, the image input unit 2 is a standard (for example, USB (Universal Serial Bus) or SCSI (Small Computer System Interface)) that connects the digital video camera, personal computer, recording device, and the like to the moving image editing apparatus 1 so that data communication is possible. Or the like (wireless connection standard such as Bluetooth (registered trademark)). Also, moving image data recorded on a recording medium (for example, various flash memories, CD (Compact Disk), DVD (Digital Versatile Disc, Digital Video Disc)) may be input to the moving image editing apparatus 1. In this case, the image input unit 2 is configured using a device (for example, a flash memory reader, a CD drive, or a DVD drive) that reads data from a recording medium.
In addition, the moving image editing apparatus 1 is included in various apparatuses (for example, a PDA (Personal Digital Assistant) or a mobile phone) including an image capturing apparatus such as a digital video camera or an image capturing apparatus such as a digital video camera. May be input to the moving image editing apparatus 1. In this case, the image input unit 2 is an interface for causing the moving image editing apparatus 1 to input moving image data picked up by an image pickup device such as a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) sensor. It may be configured as. Further, the image input unit 2 may be configured to be able to respond to a plurality of cases described above.
<Temporary Storage Unit> The temporary storage unit 3 is configured using a storage device. Any specific technique such as a volatile storage device or a nonvolatile storage device may be applied to the storage device used for the temporary storage unit 3.
The temporary storage unit 3 stores moving image data input via the image input unit 2. Data of each frame of the moving image stored in the temporary storage unit 3 is read by the head detection unit 4, the head tracking unit 5, and the cutout unit 6. The temporary storage unit 3 holds data of each frame of a moving image that is a target of the processing until at least the processing by the head detection unit 4, the head tracking unit 5, and the cutout unit 6 is completed.
<Head Detection Unit> The head detection unit 4 reads the frame data of the moving image from the temporary storage unit 3, detects the human head from the data, and indicates the position and size of the detected head. Identify head information. The head detection unit 4 may be configured to detect the head by detecting the face by template matching using a reference template corresponding to the outline of the entire face, for example. The head detection unit 4 may be configured to detect a vertex such as a head by chroma key processing and detect the head based on the vertex. The head detection unit 4 may be configured to detect a head by detecting a region close to the skin color and detecting the region as a face. Further, the head detection unit 4 may be configured to perform learning based on a teacher signal using a neural network and detect a face-like area or a head-like area as a head. In addition, the detection processing by the head detection unit 4 may be realized by applying any existing technique.
<Head Tracking Unit> The head tracking unit 5 tracks (tracks) the head detected by the head detection unit 4 in a moving image stored in the temporary storage unit 3. Moreover, the head tracking part 5 determines the head used as the object of a tracking process according to a specific reference | standard, when the head of several people is detected from the flame | frame of a moving image. The predetermined reference is, for example, a face size, a face orientation, a face position in an image, or the like. The tracking process in the head tracking unit 5 is performed by searching for and tracking feature points included in the head (for example, feature points on the forehead, eyebrows, eyes, ears, nose, lips, etc.) around the feature points of the previous image. It can be realized by doing. Further, this tracking process may be realized by other existing technology.
<Cutout Unit> The cutout unit 6 cuts out a frame to be output according to the tracking result (that is, the head position) of the head tracking unit 5. FIG. 2 is a diagram illustrating an example of a frame position cut out by the cutout unit 6. The reference point (X, Y) is a point serving as a reference for processing by the cutout unit 6 and may be arbitrarily set without changing with time t. In FIG. 2, the reference point is the center point of the frame. The head center point (x (t), y (t)) is a point obtained from the tracking result of the head tracking unit 5 and indicates the center position of the head of the person who is the subject at a certain time t. The center point of the head is an example indicating the position of the head of the person who is the subject, and need not be the center of the head, for example, the center of the face, the center of both eyes, the position of the nose or mouth Also good. The head center vector (Xd (t), Yd (t)) is a vector indicating the head center point from the reference point. The frame center vectors (Xs (t), Ys (t)) are vectors to be calculated by the cutout unit 6, and are vectors indicating the center position of the frame that the cutout unit 6 is to cut out from the reference point. Hereinafter, an example of how to obtain the frame center vector will be described.
The cutout unit 6 obtains a frame center vector based on the positions of the reference point and the head center point. At this time, the clipping unit 6 obtains the frame center vector so that the difference between the frames of the Y-axis component (vertical direction component) of the frame center vector becomes small or becomes zero. Specifically, in the clipping unit 6, for example, the influence of the head center point on the Y-axis component of the frame center vector is smaller than the influence of the head center point on the X-axis component (horizontal direction component) of the frame center vector. Or the center vector of the frame can be determined so that the influence is zero.
In order to perform such processing, the cutout unit 6 obtains a frame center vector based on, for example, Equation 1. That is, the clipping unit 6 multiplies the Y-axis component of the head center vector by a value (k) smaller than the value (j) multiplied by the X-axis component of the head center vector or zero as described above. Processing can be realized.
When the cutout unit 6 calculates the frame center vector by the above-described processing, the cutout unit 6 determines the frame position around the point indicated by the frame center vector. The frame position can be determined by the frame center vector and the vertical and horizontal lengths of the frame. The vertical and horizontal lengths of the frame may be determined in advance or may be set by the user.
The cutout unit 6 cuts out the image in the frame determined in this way, and outputs it to the moving image storage unit 7.
<Moving Image Storage Unit> The moving image storage unit 7 is configured using a nonvolatile storage device. The moving image storage unit 7 may be provided in the moving image editing apparatus 1 or may be provided outside the moving image editing apparatus 1 as shown in FIG. The moving image storage unit 7 stores a moving image composed of frames cut out by the cutout unit 6.
For example, when the moving image editing apparatus 1 is mounted on a digital video camera, the recording medium of the digital video camera corresponds to the moving image storage unit 7. For example, when the moving image editing apparatus 1 is mounted on a personal computer, a hard disk, a CD-R, a DVD-R, or the like corresponds to the moving image storage unit 7.
[Operation Example] FIG. 3 is a flowchart illustrating an operation example of the moving image editing apparatus 1. When a moving image is input via the image input unit 2 (S01), data of each frame of the moving image is stored in the temporary storage unit 3. The moving image editing apparatus 1 performs the following processing for each input frame.
First, the cutout unit 6 acquires the head center point from the processing result of the head detection unit 4 or the head tracking unit 5 (S02). Next, the clipping unit 6 calculates a head center vector from the reference point and the head center point (S03), and calculates a frame center vector using Equation 1 (S04). Then, the cutout unit 6 cuts out a frame determined by the frame center vector (S05), and outputs moving image data constituted by the frame (S06).
[Operation / Effect] FIG. 4A is an example of a frame cut out by the conventional technique. FIG. 4B shows an example in which a moving image edited by the prior art is reproduced. FIG. 5A shows an example of a frame cut out by the cutout unit 6 of the moving image editing apparatus 1. FIG. 5B shows an example in which a moving image edited by the moving image editing apparatus 1 is reproduced.
In the prior art, the position of the frame is determined only by the position of the person's head and is independent of the position of the reference point. Therefore, as shown in FIG. 4B, the human head is always located at the approximate center of the frame.
On the other hand, in the moving image editing apparatus 1, the position of the frame is determined by the position of the human head (head center point) and the position of the reference point. At this time, the position in the vertical direction of the frame is less affected by the head center point than the position in the horizontal direction (X-axis direction), and is greatly affected by the reference point. And the position of the reference point does not move between the frames. For this reason, as shown in FIG. 5 (b), the human head is always positioned at the approximate center of the frame in the X-axis direction, but is not fixed at the approximate center in the Y-axis direction. Is displaced. That is, when a moving image composed of frames cut out by the moving image editing apparatus 1 is reproduced, the human head swings up and down (vertical direction) in accordance with the movement in the Y-axis direction. Therefore, the human head is always captured at the approximate center in the X-axis direction, so that visibility is maintained, and the human head swings up and down naturally in the Y-axis direction, so that the feeling of dynamism is not lost.
When the moving image output by the moving image editing apparatus 1 is a moving image in which, for example, a figure of a person running is captured, the following effects are also obtained. When a person runs, the person's head moves up and down in small increments. When framing is done so that the human head is always centered as in the past, the human head is surely visible, but the background shakes up and down in small increments. End up. For this reason, not only the feeling of dynamism is lost, but also a moving image that is difficult to see due to the flickering background has been generated.
On the other hand, when framing is performed by the moving image editing apparatus 1, not the background but the human head moves up and down in small increments. For this reason, it is suppressed that a background shakes in small steps and it becomes difficult to see.
[Modification] The moving image editing apparatus 1 may be configured not to include the head tracking unit 5. In this case, the head detection unit 4 may be configured to perform the head tracking process by repeatedly executing the detection process on the moving image stored in the temporary storage unit 3.
In addition, the cutout unit 6 may obtain a frame center vector based on Equation 2.
In Equation 2, Xd ′ (t) and Yd ′ (t) are values calculated in consideration of the time series change of the head center vector (Xd (t), Yd (t)). Specific examples of Xd ′ (t) and Yd ′ (t) include those obtained by taking out low-frequency components of Xd (t) and Yd (t) by Fourier transform, those obtained by moving average, and those obtained through a Kalman filter. and so on. By extracting the low-frequency component, the high-frequency component is removed, and it is possible to effectively suppress the minute vibration of the frame that causes discomfort to the viewer. In addition, a reaction delay may occur by performing such a process, but the Kalman filter can also suppress the reaction delay.
The moving image editing apparatus 1 may be mounted on an imaging apparatus having an acceleration sensor. In this case, a moving image in which camera shake is effectively suppressed is picked up by the acceleration sensor, and the moving image editing apparatus 1 can further perform framing on the moving image. For this reason, it is possible to generate a moving image in which blurring in the X-axis direction is more effectively suppressed.
It is a figure which shows the example of a functional block of a moving image editing apparatus. It is a figure which shows the example of the position of the frame which a cutout part cuts out. It is a flowchart which shows the operation example of a moving image editing apparatus. It is a figure which shows the example of the frame cut out by a prior art, and the moving image after an edit. It is a figure which shows the example of the frame which the cutout part cut out, and the moving image after editing.
DESCRIPTION OF SYMBOLS 1 Moving image editing apparatus 2 Image input part 3 Temporary memory | storage part 4 Head detection part 5 Head tracking part 6 Clipping part 7 Moving image storage part
Position acquisition means for acquiring the position of the object in each frame of the input moving image;
An image surrounded by the cutout frame by determining the position of the cutout frame in the frame based on the reference position preset in the frame and the position of the target object so that the target is contained in the cutout frame. Cutting out means for reconstructing a moving image by cutting out
The moving image editing apparatus, wherein the cutout unit determines the position of a cutout frame so that the vertical component of the cutout position is less affected by the position of the object compared to the horizontal component.
The cutout unit obtains a difference between the reference position and the position of the object, multiplies the horizontal component of the difference by a first value, and the first value for the vertical component of the difference. The moving image editing apparatus according to claim 1, wherein a position of the clipping frame is determined from both the obtained value and the reference position by multiplying a second value smaller than the second value.
The cutout means obtains a difference between the reference position and the position of the object in a plurality of frames, and takes a time-series horizontal component in consideration of a time-series change between frames for the horizontal component and the vertical component of the difference. And a time-series vertical component, the time-series horizontal component is multiplied by a first value, the time-series vertical component is multiplied by a second value smaller than the first value, and both obtained values The moving image editing apparatus according to claim 1, wherein the position of the clipping frame is determined from the reference position.
The moving image editing apparatus according to claim 3, wherein the clipping unit calculates the time-series horizontal component and the time-series vertical component by performing Fourier transform.
The moving image editing apparatus according to claim 1, wherein the object is a human head or face.
A position acquisition step in which the information processing apparatus acquires the position of the object in each frame of the input moving image;
The information processing apparatus determines the position of the cutout frame in the frame based on a reference position preset in the frame and the position of the target object so that the target object is placed in the cutout frame. A clipping step for reconstructing a moving image by clipping an image surrounded by a clipping frame;
In the clipping step, the information processing apparatus determines the position of the clipping frame so that the vertical component of the clipping position is less affected by the position of the object than the horizontal component. Method.
A position acquisition step of acquiring the position of the object in each frame of the input moving image;
An image surrounded by the cutout frame by determining the position of the cutout frame in the frame based on the reference position preset in the frame and the position of the target object so that the target is contained in the cutout frame. A clipping step for reconstructing a moving image by clipping
A program for causing the information processing apparatus to execute a process of determining a position of a cutout frame so that an influence of a position of the target object on a vertical component of a cutout position is smaller than a horizontal component in the cutout step.
In the clipping step, the information processing apparatus determines the position of the clipping frame so that the vertical component of the clipping position is less affected by the position of the object than the horizontal component. Recording media that records images edited by the method.
JP2005260625A 2005-09-08 2005-09-08 Moving image editing apparatus, moving image editing method and program Active JP4513699B2 (en)
JP2005260625A JP4513699B2 (en) 2005-09-08 2005-09-08 Moving image editing apparatus, moving image editing method and program
JP2007074503A true JP2007074503A (en) 2007-03-22
JP4513699B2 JP4513699B2 (en) 2010-07-28
ID=37935547
JP2005260625A Active JP4513699B2 (en) 2005-09-08 2005-09-08 Moving image editing apparatus, moving image editing method and program
JP (1) JP4513699B2 (en)
JP2009218807A (en) * 2008-03-10 2009-09-24 Sanyo Electric Co Ltd Imaging apparatus and image reproducing apparatus
JP2010103823A (en) * 2008-10-24 2010-05-06 Sanyo Electric Co Ltd Imaging apparatus
JP2010193476A (en) * 2010-03-19 2010-09-02 Sanyo Electric Co Ltd Imaging device and image reproducing device
JP2013240110A (en) * 2013-08-21 2013-11-28 Olympus Imaging Corp Image processing device, image processing method, and photographing method
US9111363B2 (en) 2013-03-25 2015-08-18 Panasonic Intellectual Property Management Co., Ltd. Video playback apparatus and video playback method
EP3166305A4 (en) * 2014-07-04 2017-06-28 Panasonic Intellectual Property Management Co., Ltd. Image-capturing device
JP2000261657A (en) * 1999-03-12 2000-09-22 Nippon Telegr & Teleph Corp <Ntt> Method for photographing video image for editing with object trimming function, its system and storage medium with program describing the same function stored therein
2005-09-08 JP JP2005260625A patent/JP4513699B2/en active Active
JP4513699B2 (en) 2010-07-28
US20170272646A1 (en) 2017-09-21 Photographing apparatus, method and medium using image recognition
US7639282B2 (en) 2009-12-29 Image sensing device that acquires a movie of a person or an object and senses a still image of the person or the object, and control method thereof