Patent Publication Number: US-2012039515-A1

Title: Method and system for classifying scene for each person in video

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application is a continuation of application Ser. No. 11/882,733 filed on Aug. 3, 2007, which claims the priority of Korean Patent Application No. 10-2007-0000957, filed on Jan. 4, 2007, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference. 
    
    
     BACKGROUND 
     1. Field 
     The present invention relates to a method and system for classifying a scene for each person in a video, and more particularly, to a method and system for classifying a scene for each person in a video based on person information and background information in video data. 
     2. Description of the Related Art 
     Generally, a scene is a unit between when video contents are changed. In a conventional art, scenes are classified by using low level information such as color information or edge information. 
     Specifically, shots are clustered using low level information such as color information extracted in all frames, and a scene segmentation is detected in a conventional automatic scene segmentation algorithm. However, when a person in a video moves or a camera moves, low level information changes. Accordingly, a degree of accuracy decreases. 
     Also, persons in a video are clustered using face information, and thus the persons are classified in a conventional person classification method. However, face information changes depending on poses, lighting, and the like, which causes a low accuracy. 
     Accordingly, a method and system for classifying a scene for each person in a video is required. 
     SUMMARY 
     An aspect of the present invention provides a method and system for classifying a scene for each person in a video which may provide a story overview for each person by classifying a person by a scene unit by using temporal information in video data. 
     An aspect of the present invention also provides a method and system for classifying a scene for each person in a video which may improve an accuracy of a scene segmentation detection by separating a person portion and a background in video data and using information about the person portion and the background together. 
     According to an aspect of the present invention, there is provided a method of classifying a scene for each person in a video, the method including: detecting a face within input video frames; detecting a shot change of the input video frames; extracting a person representation frame in the shot; performing a person clustering in the extracted person representation frame based on time information; detecting a scene change by separating a person portion from a background based on face extraction information, and comparing the person portion and the background; and merging similar clusters from the extracted person representation frame and performing a scene clustering for each person. 
     According to another aspect of the present invention, there is provided a system for classifying a scene for each person in a video, the system including: a face detection unit detecting a face within input video frames; a shot change detection unit detecting a shot change of the input video frames; a person representation frame extraction unit extracting a person representation frame in the shot; a person clustering unit performing a person clustering in the extracted person representation frame based on time information; a scene change detection unit detecting a scene change by separating a person portion from a background based on face extraction information and comparing the person portion and the background; and a scene clustering unit merging similar clusters from the extracted person representation frame and performing a scene clustering for each person. 
     Additional aspects and/or advantages of the invention will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the invention. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee. 
       These and/or other aspects, features, and advantages of the invention will become apparent and more readily appreciated from the following description of exemplary embodiments, taken in conjunction with the accompanying drawings of which: 
         FIG. 1  is a block diagram illustrating a configuration of a system for classifying a scene for each person in a video according to an embodiment of the present invention; 
         FIG. 2  is a diagram illustrating an example of clothes information and face information detected in a same time window according to an embodiment of the present invention; 
         FIG. 3  is a diagram illustrating an example of performing a clustering for each person according to an embodiment of the present invention; 
         FIG. 4  is a flowchart illustrating a method of classifying a scene for each person in a video according to another embodiment of the present invention; 
         FIG. 5  is a flowchart illustrating an operation of a time information-based person clustering illustrated in  FIG. 4  according to another embodiment of the present invention; 
         FIG. 6  is a flowchart illustrating an operation of a scene change detection illustrated in  FIG. 4  according to another embodiment of the present invention; and 
         FIG. 7  is a flowchart illustrating an operation of a scene clustering for each person according to another embodiment of the present invention. 
     
    
    
     DETAILED DESCRIPTION 
     Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. The embodiments are described below in order to explain the present invention by referring to the figures. 
       FIG. 1  is a block diagram illustrating a configuration of a system for classifying a scene for each person in a video according to an embodiment of the present invention. 
     Referring to  FIG. 1 , the system for classifying a scene for each person in a video  100  includes a face detection unit  110 , a shot change detection unit  120 , a person representation frame extraction unit  130 , a person clustering unit  140 , a scene change detection unit  150 , and a scene clustering unit  160 . 
     The face detection unit  110  detects a face of input video frames. Specifically, the face detection unit  110  analyzes the input video frames, and detects the face of the input video frames. 
     The shot change detection unit  120  detects a shot change within the input video frames. Specifically, the shot change detection unit  120  detects the shot change of the input video frames to segment the input video frames into a shot which is a basic unit of the video. 
     The person representation frame extraction unit  130  extracts a person representation frame in the shot. Using all person frames for a person clustering is inefficient. Accordingly, the person representation frame extraction unit  130  extracts a frame which is closest to a center frame having a greatest similarity in each cluster as the person representation frame, after performing a clustering of frames including a face in the shot. Specifically, the person representation frame extraction unit  130  extracts the frame one by one in all clusters and may set the frame as the person representation frame in the shot, since at least one person may be included in the shot. 
     The person clustering unit  140  performs the person clustering in the extracted person representation frame based on time information. When simply performing a clustering based on all person representation frames, an algorithm for various poses or lightings may not be strict. Accordingly, the person clustering unit  140  performs the person clustering by using the time information to start clustering based on various forms of each person. Specifically, as illustrated in  FIG. 2 , a single person generally wears same clothes within a similar time period in same video data, and such clothes information has a clearer difference than face information. Accordingly, the person clustering unit  140  obtains various forms of the single person by using the clothes information. 
       FIG. 2  is a diagram illustrating an example of clothes information and face information detected in a same time window according to an embodiment of the present invention. 
     A location and size of a face  211 ,  221 ,  231 ,  241 , and  251 , automatically detected in a person representation frame  210 ,  220 ,  230 ,  240 , and  250  in a shot, and a location and size of clothes  212 ,  222 ,  232 ,  242 , and  252 , extracted in the person representation frame  210 ,  220 ,  230 ,  240 , and  250 , are illustrated in  FIG. 2 . The size of clothes is determined in proportion to a size of a key person in the person representation frame  210 ,  220 ,  230 ,  240 , and  250  in the shot. 
     The person clustering unit  140  extracts clothes information from current cluster information, a current person representation frame, and a comparison person representation frame, i.e. a person representation frame to be compared. The person clustering unit  140  compares the current person representation frame and the comparison person representation frame, and determines whether the current person representation frame is similar to the comparison person representation frame as a result of the comparing. The person clustering unit  140  extends a time window when the current person representation frame is similar to the comparison person representation frame, and includes the person representation frame which has been currently compared in the current cluster information. The person clustering unit  140  sets a subsequent person representation frame as another comparison person representation frame on the time window. Also, the person clustering unit  140  determines whether the current person representation frame and the comparison person representation frame are at an end of the time window, when the current person representation frame is different from the comparison person representation frame. The person clustering unit  140  sets the subsequent person representation frame in the time window as the other comparison person representation frame, when the current person representation frame and the comparison person representation frame are not at the end of the time window. 
     A scene change detection unit  150  detects a scene change by separating a person portion from a background based on face extraction information and comparing the person portion and the background. Specifically, the scene change detection unit  150  may approximately extract a person by using the face extraction information, and thus may detect the scene change by the separating and the comparing after the person is approximately extracted. 
     The scene change detection unit  150  receives current scene information, a current shot representation frame, and a comparison shot representation frame, and extracts background information from the current shot representation frame and the comparison shot representation frame. The scene change detection unit  150  compares the current shot representation frame and the comparison shot representation frame, and determines whether the current shot representation frame is similar to the comparison shot representation frame. The scene change detection unit  150  extends the time window when the current shot representation frame is similar to the comparison shot representation frame, and marks that the comparing of the current shot representation frame is completed. The scene change detection unit  150  assigns the comparison shot representation frame to the current shot representation frame, and assigns a subsequent shot representation frame in the time window to the comparison shot representation frame. The scene change detection unit  150  marks that the comparing of the current shot representation frame is completed, when the current shot representation frame is different from the comparison shot representation frame, and determines whether comparing all frames in the time window is completed. The scene change detection unit  150  assigns a subsequent shot representation frame where the comparing is incomplete to the current shot representation frame, and assigns the subsequent shot representation frame to the comparison shot representation frame, when the comparing is incomplete. 
     A scene clustering unit  160  merges similar clusters from the extracted person representation frame and performs a scene clustering for each person. Specifically, the scene clustering unit  160  may perform the scene clustering for each person by comparing the person representation frame in the shot and merging the similar clusters according to the comparison, as illustrated in  FIG. 3 . 
     The scene clustering unit  160  receives time information-based clusters, and selects two clusters having a minimum difference value. The scene clustering unit  160  compares the minimum difference value and a threshold value, and merges the two clusters when the minimum difference value is less than the threshold value. The scene clustering detection unit  160  connects scenes including a person frame in a same cluster, when the minimum difference value is equal to or greater than the threshold value. A scene clustering method for each person is described in greater detail with reference to  FIG. 3 . 
       FIG. 3  is a diagram illustrating an example of performing a clustering for each person according to an embodiment of the present invention. 
     In operation S 1 , a scene clustering unit  160  compares a first person representation frame  310  and a second person representation frame  320 , and performs a first merge of similar clusters based on a result of the comparison. In operation S 2 , the scene clustering unit  160  compares a fifth person representation frame  350  and a sixth person representation frame  360 , and performs a second merge of similar clusters based on a result of the comparison. In operation S 3 , the scene clustering unit  160  compares a third person representation frame  330  and a seventh person representation frame  370 , and performs a third merge of similar clusters based on a result of the comparison. In operation S 4 , the scene clustering unit  160  compares the first merge and the second merge, and performs a fourth merge of similar clusters based on a result of the comparison. 
       FIG. 4  is a flowchart illustrating a method of classifying a scene for each person in a video according to another embodiment of the present invention. 
     Referring to  FIG. 4 , in operation S 410 , a system for classifying a scene for each person in a video detects a face within input video frames. Specifically, the system for classifying a scene for each person in a video analyzes the input video frames via a face detector and thereby may detect the face within the input video frames. 
     In operation S 420 , the system for classifying a scene for each person in a video detects a shot change within the input video frames. Specifically, the system for classifying a scene for each person in a video detects the shot change within the input video frames to segment the input video frames into a shot which is a basic unit of the video. 
     In operation S 430 , the system for classifying a scene for each person in a video extracts a person representation frame in the shot. Since using all person frames for a person clustering is inefficient, the system for classifying a scene for each person in a video extracts a frame which is closest to a center in each cluster as the person representation frame, after performing a clustering of frames including a face in the shot. Specifically, the system for classifying a scene for each person in a video extracts the frame one by one in all frames and may set the frame as the person representation frame in the shot, since at least one person may be included in the shot. 
     In operation S 440 , the system for classifying a scene for each person in a video performs the person clustering in the extracted person representation frame based on time information. When simply clustering based on all person representation frames, an algorithm for various poses or lightings may not be strict. Accordingly, the system for classifying a scene for each person in a video performs the person clustering by using the time information to start clustering based on various forms of each person. Specifically, a single person generally wears the same clothes within a similar time period in the same video data, and such clothes information has a clearer difference than face information. Accordingly, the system for classifying a scene for each person in a video obtains various forms of the single person by using the clothes information. An operation of a time information-based person clustering is described in greater detail with reference to  FIG. 5 . 
       FIG. 5  is a flowchart illustrating an operation of a time information-based person clustering illustrated in  FIG. 4  according to another embodiment of the present invention. 
     Referring to  FIG. 5 , in operation S 501 , the system for classifying a scene for each person in a video receives current cluster information, a current person representation frame, and a comparison person representation frame. The comparison person representation frame is a person representation frame to be compared. 
     In operation S 502 , the system for classifying a scene for each person in a video extracts clothes information of each of the current person representation frame and the comparison person representation frame. Specifically, the system for classifying a scene for each person in a video may extract the clothes information by referring to the location and size of the face from the face information as illustrated in  FIG. 2  to reduce a time to extract clothes information. 
     In operation S 503 , the system for classifying a scene for each person in a video compares the current person representation frame and the comparison person representation frame. Specifically, the system for classifying a scene for each person in a video adds a comparison value of color information corresponding to the clothes information and a weight of a comparison value corresponding to the face information, when comparing. 
     In operation S 504 , the system for classifying a scene for each person in a video determines whether the current person representation frame is similar to the comparison person representation frame, as a result of the comparing. 
     In Operation S 505 , when the current person representation frame is similar to the comparison person representation frame, the system for classifying a scene for each person in a video extends a time window T fw . Specifically, when the current person representation frame is similar to the comparison person representation frame, the system for classifying a scene for each person in a video resets the time window T fw  from a present point in time, since a same person exists up to the present point in time. 
     In operation S 506 , the system for classifying a scene for each person in a video includes the comparison person representation frame which has been currently compared in the current cluster information. Specifically, the system for classifying a scene for each person in a video includes the comparison person representation frame, which has been compared with the current person representation frame, in the current cluster information. 
     In operation S 507 , the system for classifying a scene for each person in a video sets a subsequent person representation frame in the time window T fw  as other comparison person representation frame, and performs operation S 502 . Specifically, the system for classifying a scene for each person in a video continues to compare using the subsequent person representation frame in the time window T fw . 
     In operation S 508 , when the current person representation frame is different from the comparison person representation frame, the system for classifying a scene for each person in a video determines whether the current person representation frame and the comparison person representation frame are at an end of the time window T fw . Specifically, when the current person representation frame is different from the comparison person representation frame, the system for classifying a scene for each person in a video determines whether the all frames in the time window T fw  are compared by using a result of the determining whether the current person representation frame and the comparison person representation frame are at the end of the time window T fw . 
     In operation S 509 , when the current person representation frame and the comparison person representation frame are at the end of the time window T fw , the system for classifying a scene for each person in a video moves to a subsequent cluster and performs a time information-based person clustering for the subsequent cluster, since all person representation frames corresponding to a current cluster are extracted. 
     In operation S 510 , when the current person representation frame and the comparison person representation frame are not at the end of the time window T fw , the system for classifying a scene for each person in a video sets the subsequent person representation frame as the comparison person representation frame, and performs operation S 502 , since the all person representation frames corresponding to the current cluster are not detected. 
     In operation S 450 , the system for classifying a scene for each person in a video detects a scene change by separating a person portion from a background based on face extraction information and comparing the person portion and the background. Specifically, the system for classifying a scene for each person in a video may approximately extract a person by using the face extraction information, and thus may detect the scene change by the separating and the comparing after the person is approximately extracted. A scene change detection operation is described in greater detail with reference to  FIG. 6 . 
       FIG. 6  is a flowchart illustrating an operation of a scene change detection illustrated in  FIG. 4  according to another embodiment of the present invention. 
     Referring to  FIG. 6 , in operation S 601 , the system for classifying a scene for each person in a video receives current scene information, a current shot representation frame P f , and a comparison shot representation frame C f . 
     In operation S 602 , the system for classifying a scene for each person in a video extracts background information of the current shot representation frame P f  and the comparison shot representation frame C f . The background information is information about a pixel of another location excluding a face location and a clothes location. 
     In operation S 603 , the system for classifying a scene for each person in a video compares the current shot representation frame P f  and the comparison shot representation frame C f . Specifically, the system for classifying a scene for each person in a video adds the comparison value of the color information corresponding to the clothes information and the weight of the comparison value corresponding to the face information, when comparing. Also, when comparing the background information, a normalized color histogram, and a hue, saturation, value (HSV) are used. 
     In operation S 604 , the system for classifying a scene for each person in a video determines whether the current shot representation frame P f  is similar to the comparison shot representation frame C f , as a result of the comparing. 
     In operation S 605 , when the current shot representation frame P f  is similar to the comparison shot representation frame C f , the system for classifying a scene for each person in a video extends a time window T sw . Specifically, the system for classifying a scene for each person in a video resets the time window T sw  to extend a scene again, since a same scene is continued up to a point in time when the current shot representation frame P f  is similar to the comparison shot representation frame C f . 
     In operation S 606 , the system for classifying a scene for each person in a video marks that the comparing of the current shot representation frame P f  is completed, and sets the comparison shot representation frame C f  as the current shot representation frame P f . 
     In operation S 607 , the system for classifying a scene for each person in a video sets a subsequent shot representation frame in the time window T sw  as a comparison shot representation frame (*C f ?), and performs operation S 602 . Specifically, the system for classifying a scene for each person in a video continues to compare using the subsequent shot representation frame in the time window T sw . 
     In operation S 608 , when the current shot representation frame P f  is different from the comparison shot representation frame C f , the system for classifying a scene for each person in a video marks that the comparing of the current shot representation frame P f  is completed. 
     In operation S 609 , the system for classifying a scene for each person in a video determines whether comparing all frames in the time window T sw  is completed. 
     In operation S 610 , when the comparing all frames in the time window T sw  is completed, the system for classifying a scene for each person in a video determines a shot, which is examined last and determined to be a similar shot, as a last shot of a current scene, since all shots corresponding to the current scene are detected. Also, the system for classifying a scene for each person in a video performs a detection operation of a subsequent scene. 
     In operation S 611 , when the comparing is incomplete, the system for classifying a scene for each person in a video sets a subsequent shot representation frame where the comparing is incomplete as the current shot representation frame P f , and sets the subsequent shot representation frame as the comparison shot representation frame C f . Also, the system for classifying a scene for each person in a video performs operation S 602 . 
     In operation S 460 , the system for classifying a scene for each person in a video merges similar clusters from the extracted person representation frame and performs the scene clustering for each person. Specifically, the system for classifying a scene for each person in a video may perform the scene clustering by comparing and merging as illustrated in  FIG. 3 . An operation of a scene clustering for each person is described in greater detail with reference to  FIG. 7 . 
       FIG. 7  is a flowchart illustrating an operation of a scene clustering for each person according to another embodiment of the present invention. 
     Referring to  FIG. 7 , in operation S 701 , the system for classifying a scene for each person in a video receives time information-based clusters. 
     In operation S 702 , the system for classifying a scene for each person in a video selects two clusters having a minimum difference value from difference values from among all clusters. Specifically, the difference values of all clusters may be compared using an average value of each cluster. Also, the minimum difference value may be used after comparing all objects of a corresponding cluster and all objects of a comparison cluster. 
     In operation S 703 , the system for classifying a scene for each person in a video compares the minimum difference value and a threshold value and determines whether the minimum difference value is less than the threshold value. 
     In operation S 704 , when the minimum difference value is less than the threshold value, the system for classifying a scene for each person in a video merges the two clusters, as illustrated in  FIG. 3 , since the two clusters include a similar person. Also, the system for classifying a scene for each person in a video performs operation S 702 . 
     In operation S 705 , when the minimum difference value is equal to or greater than the threshold value, the system for classifying a scene for each person in a video connects scenes including a person frame in a same cluster. Specifically, the system for classifying a scene for each person in a video determines that all clustering are completed when the minimum difference value is equal to or greater than the threshold value. Also, when connecting the scenes including a same person, the operation of a scene clustering for each person is completed. Each scene may be included in many clusters since various persons may exist in a single scene. 
     The method and system for classifying a scene for each person in a video according to the above-described exemplary embodiments of the present invention may be recorded in computer-readable media including program instructions to implement various operations embodied by a computer. The media may also include, alone or in combination with the program instructions, data files, data structures, and the like. Examples of computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD ROM disks and DVD; magneto-optical media such as optical disks; and hardware devices that are specially configured to store and perform program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory, and the like. The media may also be a transmission medium such as optical or metallic lines, wave guides, etc. including a carrier wave transmitting signals specifying the program instructions, data structures, etc. Examples of program instructions include both machine code, such as produced by a compiler, and files containing higher level code that may be executed by the computer using an interpreter. The described hardware devices may be configured to act as one or more software modules in order to perform the operations of the above-described exemplary embodiments of the present invention. 
     A method and system for classifying a scene for each person in a video according to the above-described embodiments of the present invention may provide a story overview for each person by classifying a person by a scene unit by using temporal information in video data. 
     Also, a method and system for classifying a scene for each person in a video according to the above-described embodiments of the present invention may improve an accuracy of a scene segmentation detection by separating a person portion and a background in video data and using information about the person portion and the background together. 
     Also, a method and system for classifying a scene for each person in a video according to the above-described embodiments of the present invention may replay for each person in video data, and thereby may enable a user to selectively view a scene including a person that the user likes. 
     Also, a method and system for classifying a scene for each person in a video according to the above-described embodiments of the present invention may classify a person by a scene unit, which is a story unit in video data, and thereby may improve a scene classification accuracy and enable a scene-based navigation. 
     Also, a method and system for classifying a scene for each person in a video according to the above-described embodiments of the present invention may perform a video data analysis more easily by improving a scene classification accuracy in video data. 
     Although a few embodiments of the present invention have been shown and described, the present invention is not limited to the described embodiments. Instead, it would be appreciated by those skilled in the art that changes may be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.