Masking in video stream

Methods and devices for combining a mask with a selectively progressing video stream may include receiving a selection of at least one mask with a mask zone that obscures at least a portion of the video stream. The methods and devices may include receiving a selection to emplace the at least one mask at a first location within the video stream. The methods and devices may include receiving a selection to enable a tracking icon to move the at least one mask to a second location within the video stream while the video stream progresses. The methods and devices may include generating a combined output of the video stream and the selective emplacement and movement of the at least one mask during the video stream progression.

BACKGROUND

Masking is used to obscure portions of a video image stream, such as a video file stored in a computer-readable memory. For various reasons, it may be necessary to mask certain portions of a video to obscure an object. For example, security footage may include private information, such as license plates and faces, that need to be obscured to allow publishing in a manner that would otherwise violate privacy. With the advent of body cameras, this need raised an increasing concern, since such cameras can capture a large number of private individuals without their consent, for example. When the footage is used in a public manner, it is often necessary to obscure people's faces, address markers, or other objects, for privacy concerns.

To this end, a mask can be added to the video over the object to be obscured. Currently, such masking is generally emplaced frame-by-frame manually. Automated object (e.g., face or license plate) recognition algorithms exist; however, these systems do not have enough accuracy and reliability to ensure complete coverage in every frame. Moreover, among other problems in the related art, certain objects may be hard or impossible to detect automatically, and it may be necessary at times to obscure objects unknown to such algorithms or portions of an image stream that have poorly defined boundaries.

Thus, there remains an unmet need for improved image stream masking technologies.

SUMMARY

One example implementation relates to a method for combining a mask with a selectively progressing video stream. The method may include receiving a selection of at least one mask with a mask zone that obscures at least a portion of the video stream. The method may include receiving a selection to emplace the at least one mask at a first location within the video stream. The method may include receiving a selection to enable a tracking icon to move the at least one mask to a second location within the video stream while the video stream progresses. The method may include generating a combined output of the video stream and the selective emplacement and movement of the at least one mask during the video stream progression.

Another example implementation relates to a computer device. The computer device may include a memory to store data and instructions, a processor in communication with the memory, and a video editor application accessible via the memory and processor. The video editor application may be operable to receive a selection of at least one mask with a mask zone that obscures at least a portion of the video stream; receive a selection to emplace the at least one mask at a first location within the video stream; receive a selection to enable a tracking icon to move the at least one mask to a second location within the video stream while the video stream progresses; and generate a combined output of the video stream and the selective emplacement and movement of the at least one mask during the video stream progression.

Another example implementation relates to computer-readable medium storing instructions executable by a computer device. The computer-readable medium may include at least one instruction for causing the computer device to receive a selection of at least one mask with a mask zone that obscures at least a portion of the video stream. The computer-readable medium may include at least one instruction for causing the computer device to receive a selection to emplace the at least one mask at a first location within the video stream. The computer-readable medium may include at least one instruction for causing the computer device to receive a selection to enable a tracking icon to move the at least one mask to a second location within the video stream while the video stream progresses. The computer-readable medium may include at least one instruction for causing the computer device to generate a combined output of the video stream and the selective emplacement and movement of the at least one mask during the video stream progression.

Additional advantages and novel features relating to implementations in accordance with aspects of the present disclosure will be set forth in part in the description that follows, and in part will become more apparent to those skilled in the art upon examination of the following or upon learning by practice thereof.

DETAILED DESCRIPTION

Aspects of the present disclosure provide systems and methods for selective contemporaneous tracking of objects within progressing video streams, for example, to mask and/or highlight the objects (such masking and/or highlighting being interchangeably and/or collectively referred to herein by masking). In one example implementation, a user may use a mouse, or other input device, and a special icon superimposably displayed over progressing image frames in a video stream, which may be used to define a size and/or shape of a mask to insert over one or more objects and/or portions thereof. The user may selectively move the mask so as to follow the movement of the object in a video stream, for example, and the video stream and moving object may be combined into a combined video output. In an implementation, the system may automatically follow the objects with the mask, for example, and the video stream and moving objects may be combined into a combined video output. When the combined video output is then played, the object may appear as being hidden behind the mask throughout a portion or all of the outputted video stream. The systems and methods provide a mode to record such masks positions from frame to frame and to maintain the metadata associated with the mask positions in a database for further playback or export operations where the masking objects may be required.

Masking may be required in various situations, for example, when a video sequence is to be released to the public with the aim to gather additional information about a crime, or when the information (e.g., license plate) shown in a video sequence must be kept private by law enforcement or other parties for privacy reasons, whether the video is released to the public or not. In such situations, certain portions of the video, also interchangeably referred to herein as “objects” (e.g., individuals, license plates, or other identifying information or any other selected video frame portion or portions) shown in the video, may need to be masked when distributed to the public.

Highlighting may also be useful in various situations, for example, when a video sequence is to be released to the public with the aim to gather additional information about a crime and/or there is otherwise a need to bring the attention of the viewers to different elements of the video sequence by highlighting them with a bounding box around them.

The described devices and methods in accordance with aspects of the present disclosure may facilitate allowing a user to easily move a mask over a playing video while the video is progressing, so as to track with an identified object in motion and provide controllable playback of the video. As such, users without advanced video editing knowledge or experience may easily create such combined masked output videos, and less time may be expended when redacting videos, while ensuring accurate results.

Referring now toFIG. 1, an example system100for editing videos by masking and/or highlighting objects32in a video stream12may include one or more imaging devices106in communication with computer device102, such as via a network104(e.g., the Internet or an intranet) and/or a communication link with a communication component. Imaging device106may include one or more camera resources10that may capture a video, such as a video stream12and/or still picture of scene108that may be presented on a display44(FIG. 2). Display44may be located on imaging device106, computer device102and/or a device remote to computer device102. Imaging device106may include any device that may be used to capture and/or manipulate images, video, and video-related functionality, such as, but not limited to, body cameras, surveillance cameras, desktop video cameras (webcams), mobile devices, digital camcorders, still image cameras that support video streaming, or any device having one or more camera resources10.

Imaging device106may capture one or more image frames18in a video stream12of a plurality of objects32up tom objects (where m is a positive number) of scene108. Imaging device106may transmit video stream12to computer device102. In an example implementation, imaging device106may automatically transmit video stream12to computer device102. For example, a body worn camera and/or an in vehicle surveillance system may automatically transmit video streams12captured by the body worn camera and/or vehicle surveillance system to computer device102. In addition, a user of system100may upload to or otherwise access one or more video streams12via computer device102. For example, a user may select one or more video streams12to upload to computer device102. Example formats of video streams12may include, but are not limited to, G64, MP4, AVI, ASF, MOV, WMV, WebM, Ogg, Wav, jpg, jpeg, png, gif, and tiff.

Computer device102may include any mobile or fixed computer device, which may be connectable to a network. Computer device102may be or include, for example, a computer device, such as a desktop or laptop or tablet computer, a cellular telephone, a gaming device, a mixed reality or virtual reality device, a music device, a television, a navigation system, a camera, a personal digital assistant (PDA), or a handheld device, or any other computer device having wired and/or wireless connection capability with one or more other devices.

Computer device102may store the received video streams12in one or more files16. For example, computer device102may store related video streams12in a single file16. Related video streams12may include, but are not limited to, video streams of a same event, video streams of a same location, video streams received from the same imaging device106, and/or video streams of a same object. As such, one or more video streams12may be stored in a respective file16. One example use case may include creating a case file16for a crime and/or incident with several video streams12from one or more imaging devices106capturing the crime and/or incident from differing vantage points.

Computer device102may include a video editor application14executed by processor49and/or memory51of computer device102. Memory51may be configured for storing data and/or computer-executable instructions defining and/or associated with video editor application14, and processor49may execute video editor application14. An example of memory51can include, but is not limited to, a type of memory usable by a computer, such as random access memory (RAM), read only memory (ROM), tapes, magnetic discs, optical discs, volatile memory, non-volatile memory, and any combination thereof. An example of processor49can include, but is not limited to, any processor specially programmed as described herein, including a controller, microcontroller, application specific integrated circuit (ASIC), field programmable gate array (FPGA), system on chip (SoC), or other programmable logic or state machine.

Video editor application14may provide a user interface that may display outputs and/or accept inputs and/or commands from a user of system100. The user interface may provide a user the ability to view and/or edit video streams12. For example, a user may use video editor application14to mask selected individuals and/or objects of interest in a progressing video stream12, while, for example, avoiding a need to stop at each frame in order to redact content. In addition, a user may use video editor application14to redact audio in progressing video stream12.

In one example, a user may open video editor application14by opening a file16and selecting a menu icon to redact a selected video stream12. Upon selecting the icon to redact a video stream12, video editor application14may open a user account. Another example may include a user opening video editor application14by directly accessing a user account for video editor application14. Different user accounts may have varying levels of permission that may be granted to users. For example, permission levels may define access to available functionality and restrict specific users to view-only or edit-only rights. Moreover, permission levels may restrict access to specific video streams12.

Video editor application14may include a video editor component20that may make a video stream copy22of the selected video stream12before performing any redaction and/or editing of the video stream12. A user may have multiple video stream copies22with different redactions created for the original video stream12. As such, any changes and/modifications made during the editing process may be made to video stream copy22and the original video stream12may remain unchanged.

A video replay component42may display a video stream copy22for a user. A user may select to pause video stream copy22when one or more objects33are identified in the image frame18that are selected to be obscured and/or blurred by a mask. For example, a user may identify a face of a selected individual and/or a license plate of a vehicle to be obscured and/or blurred in the image frame18. In addition, video replay component42may present one or more image frames18of video stream copy22on a display. A user may select an image frame18to edit when one or more objects33are identified in the image frame18that may need to be obscured and/or blurred.

Once the video stream copy22is paused, for example, video editor application14may allow a user to select one or more masks34to insert into video stream copy22. Masks34may be added to the video stream copy22to obscure and/or blur one or more identified objects33in the scene108displayed in video stream copy22. For example, a user may select an icon to add a new mask34to the image frame18. The user may specify a shape (e.g., rectangle, square, triangle, circle, and oval) of the mask34. Each mask34selected by the user may be the same shape and/or a different shape may be used for differing masks34. As such, a variety of masking shapes may be used in a single image frame18for masks34.

In addition, a mask zone35may be defined for each mask34selected. The mask zone35may identify the mask size52and/or mask position50in an image frame18of video stream copy22. For example, a user may move the mask zone35to a location covering the one or more objects32in the image frame18and may selectively adjust the size of the mask zone35to obscure the one or more objects32, for example, by selecting a tracking icon43. The mask size52and position50may be altered as the video stream progresses. For example, the user may also resize mask size52without pausing the video, such as by selecting one or more predetermined shortcut keys, to increment and decrement the width and height of the mask34by a predetermined amount. One example may include a user using up and down arrow keys to adjust the height of the mask34(e.g., an up arrow to increase the height and a down arrow key to decrease the height). In addition, a user may use the right and left arrow keys to adjust the width of the mask34.

The mask zone35may also be used on any region in an image frame18, whether depicting a particular object32or not. For example, in a moving pan of a particular scene108, a mask zone35may be used to outline or obscure a portion of each frame that represents a particular location in the scene108displayed in the video stream as the relative location of changes with changes in frame perspective over time, e.g., such location may serve as a reference location.

In an example implementation, video editor application14may include an object identifier component45that may automatically identify one or more objects33in the video stream copy22to obscure and/or blur and may automatically select one or more masks34to insert into the video stream copy22over the identified objects33. One or more facial recognition systems or other object recognition system, for example, may be used to identify one or more objects33to obscure and/or blur in the video stream copy22. For example, mask34may be automatically applied on all faces detected in video stream copy22using one or more facial recognition systems. A user may edit and/or modify the one or more masks34automatically inserted into the video stream copy22. For example, a user may remove masks34from persons of interest and/or add masks34to content not captured by the automatic process (e.g., partial faces, non-face related information, and/or sides of faces). By using an automatic process to perform an initial identification, as well as to persistently track such identified portions during video progression, the editing time to redact a video may be reduced.

When a mask34is added to an initial image frame18, a mask duration36timeline may be may be initiated for presentation to the user interface corresponding to a duration of when the mask34is presented in the video stream copy22. For example, a mask duration36may begin at a mask start time38that corresponds to a time marker in the video stream copy22when the mask34is first inserted into an image frame18within the video stream copy22. In addition, the mask duration36may end at an end time40that corresponds to a time marker in the video stream copy22when the mask34is removed from an image frame18. For each mask34inserted into the image frame18, a different timeline for mask duration36may be presented. As such, a user may be able to visually determine time frames that different masks34are inserted in the video stream copy22. Moreover, a user may modify a mask duration36by moving the presented timeline. For example, a user may extend and/or shorten the mask duration36by moving the start time38and/or end time40of the mask duration36.

The interface may include a video replay component42that may receive an input via a tracking icon43to start replaying the video stream copy22. For example, a user may select the same tracking icon43used to resize and/or move the mask34in order to start replaying the video stream copy22. As the video stream copy22replays, a user may use the tracking icon43to move the mask34and follow any movement objects32may have during playing of the video stream copy22. As the mask34moves, the mask zone35associated with mask34may be updated so that the mask position50and/or mask size52may be tracked over several image frames18of the video stream copy22. Thus, a user may easily obscure one or more objects32over several image frames18while the video stream copy22continues to play at a specified speed (e.g., a selected frame rate per second for the video playback). In an implementation, a user may adjust the selected video playback speed by selecting one or more predetermined shortcut keys, to increment and/or decrement the playback speed by a predetermined amount. For example, a user may use up and down arrow keys to adjust the playback speed (e.g., an up arrow to increase the speed and a down arrow key to decrease the speed).

Video editor component20may thereby automatically generate edited image frames24for each image frame18where the mask(s)34are present and use the one or more masks34to obscure and/or blur identified objects33. For example, video editor component20may apply a blurring effect, pixilation effect, and/or black box to modify the edited image frame24corresponding to the mask position50and mask size52so that identified objects33are not present in the edited image frame24. As such, identified objects33may not be retrievable from video stream copy22once obscured and/or blurred by video editor component20.

Video editor component20may also generate and save metadata48for each edited image frame24. Metadata48may track any changes and/or modifications made to mask(s)34that may occur between edited images frames24. Metadata48may include, but is not limited to, mask position50, mask size52, and/or mask duration36. For example, metadata48may include a location vector associated with mask(s)34that may be used to track the various mask positions50of mask34.

One example use case may include a user first selecting the tracking icon43associated with mask34and then maintain persistent selection of an input device, such as holding a mouse button or maintaining a finger placement on a touchscreen, to start the masking process for a progressing video. As long as the user selects the tracking icon43, the video continues to progress (i.e., play). The user may move the input device and/or finger, and the mask34may correspondingly move so as to follow the same path as the input device. Thus, as identified object33moves, the user may follow with the input device and/or finger the moving identified object33. The user may also resize mask34without pausing the video, such as by selecting one or more predetermined shortcut keys, to increment and decrement the width and height of the mask34by a predetermined amount (e.g., 5%).

Once the user releases the tracking icon43, for example, by releasing the input device and/or removing a finger from the tracking icon43on a touch screen, the video playback may stop. In addition, the end time40of the mask duration36may correspond to the time within the video stream that the releasing the input device and/or removing a finger from the tracking icon43occurs. Video editor application14may track and record the movement of mask34, for example, in metadata48. When the user replays video stream copy22with the applied metadata, the user may see that, as the video copy plays, mask34follows the same path that the input device followed frame by frame.

At any moment during video streaming and editing operation, the user may select to click on, or otherwise select, the tracking icon43to correct and/or continue the tracking of mask34. Changes and/or modifications made to mask34may overwrite the mask position50previously recorded for mask34, if any, and/or a new recorded mask position50may be added to metadata48for mask34. Thus, every time a new mask position50is recorded for mask34, the mask position50may persist from the current time in the video stream copy22until the end time40of the mask duration36for mask34, or, for example, until a different position for mask34may be set, whichever may come first.

At any moment during video streaming and editing operation, the user may pause the video playback to resize mask34. For example, a user may resize mask34by dragging one of its sides. The user may also resize mask34without pausing the video playback by pressing shortcut keys to increment and/or decrement the width and height of mask34by a predetermined amount (e.g., 5%). The new width and height of mask34may be recorded in metadata48, and mask34may retain these new dimensions from the current time in the video stream copy22until the end time40of the mask duration36for mask34or until a different mask size52may be set, for example, whichever comes first.

Thus, a user may easily obscure one or more objects32over several edited image frames24while the video stream copy22continues to play at a specified speed (e.g., a selected frame rate which to display the edited image frames24), without having to pause the video stream copy22to see each frame and perform the editing separately on each frame, for example.

In an example implementation, a user may also select an audio file25associated with the video stream12to edit and/or modify. A user may want to remove a portion of the audio file25and/or modify a portion of the audio file25, for example, so as to render the audio information unrecognizable. For example, a user may want to remove or modify audio associated with a specific speaker in the video stream12. In addition, a user may want to remove and/or modify background noise that may identify where the video was captured and/or identify individuals located in the background of the video stream. Video editor application14may include an audio editor component46that allows a user to modify and/or change an audio file25and generate edited audio47.

For example, a user may apply one or more masks34to obscure selected audio portions in the audio file25. Each mask34may correspond to a different portion of the audio file25to remove and/or obscure the selected audio portion. The mask position50may indicate where in the audio file25the mask34may be inserted in order to remove the audio and/or obscure the audio to thereby render it unrecognizable. Audio editor component46may generate an output of edited audio47with the one or more masks34inserted into the audio. File16may associate the edited audio47with one or more video stream copies22and/or video streams12.

Referring now toFIG. 2, illustrated is an example of various aspects of processing video stream copy22. Video editor component20may receive a video stream12with one or more (e.g., 1 to r, where r is an integer) image frames18. As discussed inFIG. 1, one or more masks34may be selected for addition to a video stream copy22, and video editor component20may generate one or more (e.g., 1 to n, where n is an integer) edited image frames24with the selected masks34inserted therein.

Each mask34inserted into video stream copy22may include a mask duration36that corresponds to a time period when a mask34is present in edited image frames24. For example, the mask duration36may occur after a start time38(e.g., time T0inFIG. 2) corresponding to a first edited image frame24where mask34is present (e.g., Edited Image Frame1inFIG. 2) and continues until an end time40(e.g., time TpinFIG. 2) corresponding to a last edited image frame24in which the mask34occurs (e.g., Edited Image FrameninFIG. 2). When the mask duration36is completed (e.g., at Edited Image Framen), video editor component20may cease the generation and insertion of edited image frames24into video stream copy22, and thus video stream copy22may again include only image frames18from video stream12. As such, video stream copy22may include a combined mix of edited image frames24and image frames18.

In addition, each edited image frame24where mask34occurs may also include metadata48associated with mask34. Metadata48may include, but is not limited to, information regarding mask zone35, such as the position50of mask34in the edited image frame24, the size of mask34, and/or information relating to time54corresponding to a time marker in the video stream copy22where mask34is located. When more than one mask34is present in an edited image frame24, metadata48may be captured for each mask34inserted into the edited image frame24. Moreover, if the mask zone35changes between edited image frames24, the metadata48information may be updated. As such, the metadata48information48may track any changes to mask position50and/or mask size52between edited image frames24. For example, metadata48may include a location vector associated with mask(s)34that may be updated when the various mask positions50of mask34change.

Video editor component20may apply a blurring effect, pixilation effect, and/or black box to modify the edited image frame24corresponding to the mask position50and/or mask size52so that identified objects33are not present in the edited image frame24. As such, identified objects33may not be retrievable from video stream copy22once obscured and/or blurred by video editor component20. For example, video editor component20may modify image information in the video stream to add the blurring effect, pixilation effect, and/or black bock corresponding to the mask position and/or mask size52so that the identified objects33are not present in the edited image frame24. In addition, video editor component20may apply the blurring effect, pixilation effect, and/or black bock corresponding to the mask position and/or mask size52based on the metadata48associated with the mask34.

Video stream copy22may be presented on display44when a user selects to replay the video stream copy22. In addition, video stream copy22may be transmitted to other systems for further processing.

Referring now toFIG. 3, an example method300is illustrated for masking objects32(FIG. 1) that may be executed by video editor application14(FIG. 1) on computer device102(FIG. 1).

At302, method300may include determining whether a pause occurred in playing the video stream. For example, as shown inFIG. 1, a video replay component42may play video stream copy22for viewing by a user. A user may select to pause video stream copy22, for example, to view a scene more closely and/or when one or more objects33are identified in the image frame18that may need to be obscured and/or blurred. In addition, video replay component42may present one or more image frames18of video stream copy22on a display. A user may select an image frame18to edit when one or more objects33are identified in the image frame18that may need to be obscured and/or blurred.

At304, method300may include receiving identification of at least one object to obscure. For example, as shown inFIG. 1, a user may identify a face of an individual and/or a license plate of a vehicle that may need to be obscured and/or blurred in the image frame18. In an example implementation, video editor application14may automatically identify one or more objects33in the video stream copy22to obscure and/or blur. For example, one or more facial recognition algorithms or other object recognition system may be used to identify one or more objects33to obscure and/or blur in the video stream copy22.

At306, method300may include receiving a selection of at least one mask to insert into the video stream to obscure the at least one object. For example, as shown inFIG. 1, once the video stream copy22is paused, video editor application14may allow a user to select one or more masks34to insert into video stream copy22. For example, a user may selectively emplace mask34at a first location within the video stream (e.g., a first location within an image frame18of the video stream). Masks34may be added to the video stream copy22to obscure and/or blur one or more identified objects33in the scene108captured in video stream copy22. For example, a user may select an icon to add a new mask34to the image frame18. The user may specify a shape (e.g., rectangle, square, triangle, circle, or oval) of the mask34. Each mask34selected by the user may be the same shape and/or a different shape from other masks34. As such, a variety of shapes may be used in a single image frame18for a plurality of masks34.

In addition, a mask zone35may be defined for each mask34selected. The mask zone35may identify the mask size52and/or mask position50in an image frame18of video stream copy22. For example, a user may move the mask zone35to a location covering the one or more objects32in the image frame18and may adjust the size of the mask zone35to obscure the one or more objects32, for example, by selecting a tracking icon43. The mask zone35may also be used on any region in an image frame18, whether depicting a particular object32or not. Thus, each mask34may be associated with a persistent subportion of the video stream. The persistent subportion of the video stream may comprise at least a portion of a particular object32visible within the visible video stream. For example, in a moving pan of a particular scene108, a mask zone35may be used to outline or obscure a portion of each frame that represents a particular location in the scene108being video streamed as location position moves within the frame, e.g., using the location as a reference point. As such, the mask zone35may be used to outline or obscure at least a portion of the video stream.

In an example implementation, video editor application14may automatically select one or more masks34to insert into the video stream copy22over the identified objects33. For example, automatic face detection tools may be used to automatically generate masks and may selectively emplace the masks at a first location within the video stream and selectively move the masks to a second location within the video stream to follow and blur faces in the video when movement of the faces occurs. A user may edit and/or modify the one or more masks34automatically inserted into the video stream copy22using, for example, tracking icon43to adjust the mask zone35associated with masks34. By using automatically generated masks, time may be saved during the editing process.

At310, method300may include receiving an input to resume playing the video stream. For example, as shown inFIG. 1, the interface may include a video replay component42that may receive an input via a tracking icon43to start replaying the video stream copy22. For example, a user may select the same tracking icon43used to resize and/or move the mask34to start replaying the video stream copy22. As the video stream copy22replays, a user may use the tracking icon43to move the mask34and follow any movement that objects32may have during playing of the video stream copy22. For example, the user may selectively move mask34to a second location within the video steam when the video stream progresses to track any movement that objects32may have. As such, the movement of mask34may be linked to the movement of objects32may have when the video stream progresses. In an implementation, the movement of mask34may be automatically linked to the movement of objects32when the video stream progresses.

As the mask34moves, the mask zone35associated with mask34may be updated so that the mask position50and/or mask size52may be tracked over several edited image frames24of the video stream copy22. For example, the user may also resize mask size52without pausing the video, such as by selecting one or more predetermined shortcut keys, to increment and decrement the width and height of the mask34by a predetermined amount. One example of predetermined shortcut keys may include using up and down arrow keys to adjust the height of the mask34(e.g., an up arrow to increase the height and a down arrow key to decrease the height). In addition, predetermined shortcut keys may include using the right and left arrow keys to adjust the width of the mask34. Thus, a user may easily obscure one or more objects32over several edited image frames24while the video stream copy22continues to play at a specified speed (e.g., a selected frame rate at which to display the image frames18), without having to pause the video stream copy22to perform the editing frame by frame, for example. In an addition, a user may adjust the selected video playback speed by selecting one or more predetermined shortcut keys, to increment and decrement the playback speed by a predetermined amount. For example, a user may use up and down arrow keys to adjust the playback speed (e.g., an up arrow to increase the speed and a down arrow key to decrease the speed).

Once the user releases the tracking icon43, for example, by releasing the input device and/or removing a finger from the tracking icon43on a touch screen, the video playback may stop. In addition, the end time40of the mask duration36may correspond to releasing the input device and/or removing a finger from the tracking icon43. Video editor application14may track and record the movement of mask34, for example, in metadata48. When the user replays video stream copy22, the user may see that mask34follows the same path that the input device followed frame by frame as the video copy plays.

At314, method300may include generating a video stream copy with the at least one mask inserted into the video stream copy. Video editor component20may generate edited image frames24for each image frame18where the mask(s)34are present and used to obscure and/or blur identified objects33. For example, video editor component20may apply a blurring effect, pixilation effect, and/or black box to modify the edited image frame24corresponding to the mask position50and mask size52, so that identified objects33are not present in the edited image frame24. The identified objects33may not be retrievable from video stream copy22once obscured and/or blurred by video editor component20. For example, video editor component20may modify image information in the video stream to add the blurring effect, pixilation effect, and/or black bock corresponding to the mask position and/or mask size52so that the identified objects33are not present in the edited image frame24. In addition, video editor component20may apply the blurring effect, pixilation effect, and/or black bock corresponding to the mask position and/or mask size52based on the metadata48associated with the mask34. As such, video editor component20may generate a combined output of the video stream copy22and the selective emplacement and movement of the mask34during the video stream progression.

Video editor component20may also generate and save metadata48for each edited image frame24. Metadata48may track any changes and/or modifications made to mask(s)34, for example, that may occur between edited images frames24. Metadata48may include, but is not limited to, mask position50, mask size52, and/or mask duration36. In addition, metadata48may be retained so as to enable a user can quit and/or pause an editing session and come back later without losing the recordings previously made for a mask34.

At316, method300may optionally include adjusting a position of the at least one mask. For example, as shown inFIG. 1, a user may click on, or otherwise select, the tracking icon43to correct and/or continue the tracking of mask34. Changes and/or modifications made to mask34may overwrite the mask position50previously recorded for mask34, if any, or a new recorded mask position50may be added to metadata48for mask34. Thus, every time a new mask position50is recorded for mask34, the mask position50may stay effective from the current time in the video stream copy22until the end time40of the mask duration36for mask34or until a different position for mask34may be set, for example, whichever may come first.

The user may also pause the video playback to resize mask34and/or reposition mask34. For example, a user may resize mask34by dragging one of its sides. The user may also resize a mask by dragging it to a different position. The new X and Y coordinates of the mask position50may be recorded, and mask34may retain these new dimensions from the current time of the video until the end time40of mask34if there is not any other repositioning of the mask performed, for example. If some repositioning operations occur on mask34, then the latest operations may overwrite the previous mask position50.

The user may also resize mask34without pausing the video playback by pressing shortcut keys, for example, to increment and/or decrement the width and height of mask34by a predetermined amount (e.g., 5%). The new width and height of mask34may be recorded in metadata48, and mask34may retain these new dimensions from the current time in the video stream copy22until the end time40of the mask duration36for mask34or until a different mask size52may be set, for example, whichever comes first.

At318, method300may optionally include adjusting a mask duration when the at least one mask is visible in the video stream copy. For example, as shown inFIG. 1, a user may modify a mask duration36by moving the timeline information for the presented mask duration36. For example, a user may extend and/or shorten the mask duration36by moving the start time38and/or end time40of the mask duration36on the presented timeline.

Once a user is satisfied with the masks34created and the tracking of the identified objects33, the video editor application14may use a smoothing algorithm, for example, to avoid having a shaky trajectory for the masks34. An example of a smoothing algorithm may include, but is not limited to, removing every mask34(or changes in position thereof) through all the recordings having a change in their mask position50(e.g., X and Y coordinates) of less than 1% of their previous recording, if any.

Referring now toFIGS. 4A-4H, illustrated are example snapshots of graphical user interface screens in accordance with aspects of the present disclosure.FIG. 4Aillustrates an example user interface400where a video sequence is paused. A video timeline402representing the video sequence may indicate at404where the video sequence is paused. A user may pause the video sequence, for example, when one or more objects may be identified in the image frame that may need to be obscured and/or blurred. For example, a user may identify a face of an individual and/or a license plate of a vehicle that may need to be obscured and/or blurred in the image frame.

FIGS. 4B and 4Cillustrates example interfaces406,410with masks408,412inserted over identified objects (e.g., a face of an individual and a license plate) to hide the identified objects in the video stream sequence.

FIG. 4Dillustrates an example interface414with a tracking icon, such as a “Manual Live Tracking” button416(further illustrated inFIG. 4H). Once a user selects the tracking icon416(e.g., by touching the tracking icon416on the screen and/or pressing a mouse button), and maintains the selection of the tracking icon416, the video starts playing at the speed specified in the tracking speed menu418illustrated inFIG. 4E. In this example, the video plays at half its original speed.

As long as the user continues to select the tracking icon416, the video may continue to play. The user may move the pointer (e.g., via mouse or finger) and the mask will move so as to follow the same path as the pointer. As such, the user may follow with the pointer (e.g., mouse or finger) the moving object to be masked. The user may also resize the mask without pausing the video by pressing predetermined shortcut keys, for example, to increment and decrement the width and height to the mask by a predetermined amount (e.g., 5%).

FIG. 4Fillustrates a timeline420indicating (e.g., in blue) the length of the video and (e.g., in green) the mask duration (e.g., a length of time during which the selected mask is displayed from a start time to an end time). When a new mask is added by default, an end time may be set to the end of the video. A user may modify the end time by releasing the tracking icon416and/or moving the end of timeline420, for example.

FIG. 4Gillustrates on the right side of the video that a user may select to view a list422of the created mask with the one selected highlighted424(e.g., in green). Each line shows the mask's name, its start time and its end time. A mask selection can be toggled by clicking or otherwise selecting the line corresponding to the mask on the right side of the video player or by clicking or otherwise selecting a mask directly in the video player. When a user selects a different mask, a timeline420(FIG. 4F) may be displayed for the corresponding mask, and the tracking icon416may be selected so that the mask may be updated and/or moved.

As shown inFIG. 4H, a user is able to set the start time and the end time of the mask, which respectively correspond to the first and to the last moment within the video stream that the mask is displayed. For example, a user may drag, or otherwise move, both sides426,428of the mask timeline420under the video (e.g., in green; or the one above the video timeline402).

Referring now toFIGS. 5A-5F, additional example snapshots of user interface screens are illustrated. For example,FIG. 5Aillustrates an example user interface screen500that may be presented when video editor application14(FIG. 1) opens. For example, interface screen500may present a list502of detected faces of individuals automatically identified in a video stream using, for example, an automatic face recognition tool and/or object recognition tool. In addition, interface screen500may present a series of thumbnails506with previews of image frames of the video stream. A user may select an individual504, e.g., person3, from the list502of detected individuals. When a user selects an individual504, an image frame508with the first appearance of the selected individual504may be highlighted in the series of thumbnails506. A user may select a thumbnail associated with image frame508, and the image frame508may then be presented on the left side512of interface500as shown inFIG. 5A. In addition, a mask510may be automatically inserted into the image frame508for covering the face of the selected individual504(e.g., person3), for example. A user may select an edit mask icon514to edit the mask associated with selected individual504.

FIG. 5Billustrates an example user interface screen500when a user selects edit mask icon514. A video mask tab516may open with the mask corresponding to the selected individual504. Previously created masks518(e.g., Mask #1, Mask #2, and Mask #3) may be manually created and may be presented below the automatically created masks (e.g., Mask Person #3,504). The previously created masks518may be presented in order, for example, by start time in the video stream, with the mask corresponding to the earliest start time relative to the other start times presented first. In addition, a mask duration timeline519may be presented that identifies a duration of the mask510(e.g., by corresponding color or other identifier) and locations in the video stream where the mask510is inserted.

FIG. 5Cillustrates an example user interface screen500when a user selects a different mask520to view. The mask510associated with the selected individual504may be collapsed, and the mask522associated with newly selected mask520may be presented in the image frame. In addition, mask duration timeline519may be updated to illustrate the duration of mask522and the locations in the video stream where mask522is inserted, for example.

In an example implementation, among other things, a user may use the mask duration timeline519to view a preview of the video. For example, a user may select a start of a mask duration, and a preview of the video where the mask is first present in the video may be highlighted in the panel of thumbnails506. In addition, a user may select an end of the mask duration, and a preview of the video of the end of the mask may be highlighted in the panel of thumbnails506.

FIG. 5Dillustrates an example user interface screen500when a user selects to adjust a blur level of one or more masks in the video. For example, a user may use a blur adjustment scale524to modify the blur level of the masks in the video. In an example implementation, the user may adjust the level of blur between a 20% blur level and a 100% blur level, where the percentage indicates a level of obstruction in the image. The blur adjustment scale524may allow a user to adjust the blur level by 20% increments, for example. In another implementation, a user may be able to adjust the blur level of the masks when viewing the final video stream copy.

FIG. 5Eillustrates an example user interface screen500when a user selects to edit audio associated with a video. A user may select an audio icon525to receive a transcript526of the audio associated with the video. The transcript526may identify various speakers and associate the audio with various speakers (e.g., by color or other identifier). For example, the system may convert the audio to text using one or more known methods to provide a transcript526of the captured audio. A user may select audio528to edit and/or redact. Once a user selects a portion of the audio528to edit, a user may select an audio mask icon530to apply a mask to the selected audio528.

FIG. 5Fillustrates an example user interface screen500when a user selects the audio mask icon530. User interface screen500may present a list of audio masks532that have been created. A user may select an audio mask534to edit (e.g., audio mask #1). When a user selects audio mask534to edit, an audio mask timeline536may be presented so that a user may easily identify where audio mask534may be inserted in the audio file. The audio mask534may be used to remove the audio from the audio file and/or modify the audio so that the audio is unrecognizable. When placing audio mask534into the video, the audio may stop one second prior to the location where the audio mask534may be inserted, for example, to aid in the placement of the audio mask534.

An example implementation will now be provided with pseudocode, which is meant to illustrate, not limit, the described methods and devices. Among other things, the pseudo-code explains in further details example logic used for the “Manual live tracking” functionality once a user activate the functionality on a mask as described in the “Step-by-step of the method” section of this document at the step ofFIG. 4D. The example implementation also explains one example of the behavior of the system when the user plays back a video:If the user activates the «manual live tracking» functionality on a mask,While the user keeps the functionality activated,Start playing the video at a playback speed adjustable by the user (i.e., by default at half the original speed of the video).At every predetermined interval of time Z set in the system (i.e., every 1/30 of a second)Check if the last recording for the mask does not indicate the same X and Y positions for the mask as the ones of the user pointer right now (plus some deltas for the X and Y positions of the pointer to take into account the fact the user pointer is not positioned at the center of the mask) OR check if there is no any last recording for the mask,If that is the case, create a new recording for the mask indicating that at the current time, the X and Y positions of the mask changed to the ones of the user pointer right now (plus some deltas for the X and Y positions of the pointer to take into account the fact the user pointer is not positioned at the center of the mask).Check if the last recording for the mask does not indicate the same dimensions for the mask (width and height) as the ones specified by the user right now OR check if there is not any last recording for the mask,If that is the case, create a new recording for the mask indicating that, at the current time, the width and height of the mask changed to the ones the user specified.If the user plays back the video,At every predetermined interval of time Z set in the system (i.e., every 1/30 of a second)For every maskCheck if the mask is to be displayed at the current time (if its start time is smaller than the current time and its end time is greater than the current time).If that is the case, find the latest recording for the mask that has been recorded at the current time of the video or earlier. Display the mask over the video with the dimensions and at the X and Y positions specified in the latest recording.If that is not the case, hide the mask if it was displayed.

Aspects of the present disclosure may be implemented using hardware, software executed by a processor, or a combination thereof and may be implemented in one or more computer systems or other processing systems. For example, the user interfaces illustrated inFIGS. 4A-4H and 5A-5Fmay be implemented as a computer system or processing system. Also, the video editor application14(FIG. 1) may be implemented as a computer system or processing system. In an aspect of the present disclosure, various features may be directed toward one or more computer systems capable of carrying out the functionality described herein. An example of such a computer system600is shown inFIG. 6.

Computer system600includes one or more processors, such as processor604. The processor604is coupled to a communication infrastructure606(e.g., a communications bus, cross-over bar, or network). Various software aspects are described in terms of this example computer system. After reading this description, it will become apparent to a person skilled in the relevant art(s) how to implement aspects hereof using other computer systems and/or architectures.

Computer system600may include a display interface602that forwards graphics, text, and other data from the communication infrastructure606(or from a frame buffer not shown) for display on a display unit630. Computer system600may include a main memory608, preferably random access memory (RAM), and may also include a secondary memory610. The secondary memory610may include, for example, a hard disk drive612and/or a removable storage drive614, e.g., an optical disk drive. The removable storage drive614may read from and/or write to a removable storage unit618in a well-known manner. As will be appreciated, the removable storage unit618may include a computer readable storage medium having stored therein computer software and/or data.

Alternative aspects of the present invention may include secondary memory610and may include other similar devices for allowing computer programs or other instructions to be loaded into computer system600. Such devices may include, for example, a removable storage unit622and an interface620. Examples of such may include a removable memory chip (such as an erasable programmable read only memory (EPROM), or programmable read only memory (PROM)) and associated socket, and other removable storage units622and interfaces620, which allow software and data to be transferred from the removable storage unit622to computer system600.

Computer system600may also include a communications interface624. Communications interface624may allow software and data to be transferred among computer system600and external devices. Examples of communications interface624may include a modem, a network interface (such as an Ethernet card), a communications port, a Personal Computer Memory Card International Association (PCMCIA) slot and card, etc. Software and data transferred via communications interface624may be in the form of signals628, which may be electronic, electromagnetic, optical or other signals capable of being received by communications interface624. These signals628may be provided to communications interface624via a communications path (e.g., channel)626. This path626may carry signals628and may be implemented using wire or cable, fiber optics, a telephone line, a cellular link, a radio frequency (RF) link and/or other communications channels. As used herein, the terms “computer readable medium,” “computer program medium,” and “computer usable medium” refer generally to media such as a removable storage drive680, and/or a hard disk installed in hard disk drive670. These computer program products may provide software to the computer system600. Aspects of the present invention are directed to such computer program products.

Computer programs (also referred to as computer control logic) may be stored in main memory608and/or secondary memory610. Computer programs may also be received via communications interface624. Such computer programs, when executed, may enable the computer system600to perform the features in accordance with aspects of the present invention, as discussed herein. In particular, the computer programs, when executed, may enable the processor604to perform the features in accordance with aspects of the present invention. Accordingly, such computer programs may represent controllers of the computer system600.

Where aspects of the present invention may be implemented using software, the software may be stored in a computer program product and loaded into computer system600using removable storage drive614, hard disk drive612, or interface620. The control logic (software), when executed by the processor604, may cause the processor604to perform the functions described herein. In another aspect of the present disclosure, the system may be implemented primarily in hardware using, for example, hardware components, such as application specific integrated circuits (ASICs). Implementation of the hardware state machine so as to perform the functions described herein will be apparent to persons skilled in the relevant art(s). In yet another variation, aspects of the present disclosure may be implemented using a combination of both hardware and software.

In yet another aspect of the disclosure, the invention may be implemented using a combination of both hardware and software.

FIG. 7shows various elements of an example network700usable in accordance with aspects of the disclosure. The network700includes one or more accessors760,762(also referred to interchangeably herein as one or more “users”) and one or more terminals742,766. In one aspect of the disclosure, data for use is, for example, input and/or accessed by accessors760,762via terminals742,766, such as personal computers (PCs), minicomputers, mainframe computers, microcomputers, telephonic devices, or wireless devices, personal digital assistants (“PDAs”) or a hand-held wireless devices (e.g., wireless telephones) coupled to a server743, such as a PC, minicomputer, mainframe computer, microcomputer, or other device having a processor and a repository for data and/or connection to a repository for data, via, for example, a network744, such as the Internet or an intranet, and/or a wireless network, and couplings745,746,764. The couplings745,746,764may include, for example, wired, wireless, or fiberoptic links. In another aspect of the disclosure, the method and system of the disclosure may include one or more features that operate in a stand-alone environment, such as on a single terminal.

While implementations of various aspects of the present disclosure have been described in connection with examples thereof, it will be understood by those skilled in the art that variations and modifications of the implementations described above may be made without departing from the scope hereof. Other implementations will be apparent to those skilled in the art from a consideration of the specification or from a practice in accordance with examples disclosed herein.