Patent Description:
Due to the widespread popularity of AV content for the distribution of entertainment and news, effective management and control of legitimately created or edited content is important to the owners and distributors of that content. However, the unauthorized manipulation of AV content described above, whether performed innocently or maliciously, is typically difficult to detect. As a result, subtly altered or even substantially "fake" content may inadvertently be used in place of original AV content or authorized modifications to that original content in violation of contractual agreement or regulatory restrictions, thereby subjecting the content owners and/or distributors to potential legal jeopardy.

<CIT> discloses media classification for media identification and licensing, whereby a media content item may be received by a first processing device, a set of features of the media content item is determined, the set of features determined from the media content item is analyzed using a media classification profile comprising a first model for a first class of media content items and a second model for a second class of media content items. As a result of the analysis it may be determined whether the media content item belongs to the first class or the second class. Further reference is made to <CIT> and <NPL>.

The present invention provides a method for tracking a chain-of-manipulation of an audio-video content according to claim <NUM>, and a device with a hardware processor being adapted to execute the method according to claim <NUM>.

The dependent claims describe preferred embodiments.

Preferably, the method, wherein the one or more CoM tags further include an application tag identifying the AV processing application used for modifying the AV content.

Preferably, the method, wherein the manipulation (modification) of the AV content modifying the at least one of the video content or the audio content includes at least one of changing a contrast of a video frame, deleting a video frame, inserting a video frame, removing an object from a video frame, inserting an object into a video frame, changing colors within a video frame of the AV content, deletion of a portion of original audio content, insertion of additional audio content, mixing audio tracks, changing the audio level of an audio track, or adding metadata to an audio track.

Preferably, the method, wherein the one or more CoM tags further include at least one origination tag identifying at least one of an original author of the AV content, an original creating device of the AV content, at least one application used to create the AV content, or a creation date of the AV content.

Preferably, the method, wherein the one or more CoM tags further include at least one of a date of the manipulation of the AV content or an identification of a user performing the manipulation of the AV content.

Preferably, the method, wherein each of the plurality of segments of the AV content modified by the AV processing application comprises a sequence of video frames, and wherein the one or more CoM tags is/are inserted in a single frame of the sequence of video frames.

Preferably, the method, wherein the single frame is an I-frame of the sequence of video frames.

Preferably, the method, wherein each of the plurality of segments of the AV content modified by the AV processing application comprises a single frame of video.

Preferably, the method, wherein the one or more CoM tags is/are persistent tags that cannot be altered after insertion in each of the plurality of segments of the AV content modified by the AV processing application.

Preferably, the method, wherein the device is one of a video camera, a desktop computer, a laptop computer, a tablet computer, a gaming console, a smartphone, a smart television, or a wearable personal communication device.

According to an embodiment, the invention comprises:
A device comprising a hardware processor, and a system memory storing the CoM tagging software code and the AV processing application, the hardware processor being adapted to execute the method for tracking a chain-of-manipulation (CoM) of an audio-video (AV) content as described herein.

The following description contains specific information pertaining to implementations in the present disclosure. One skilled in the art will recognize that the present disclosure may be implemented in a manner different from that specifically discussed herein. The drawings in the present application and their accompanying detailed description are directed to merely exemplary implementations. Unless noted otherwise, like or corresponding elements among the figures may be indicated by like or corresponding reference numerals. Moreover, the drawings and illustrations in the present application are generally not to scale, and are not intended to correspond to actual relative dimensions.

The present application discloses devices and methods for use in systems for performing automated chain-of-manipulation (CoM) tracking of audio-video (AV) content that overcome the drawbacks and deficiencies in the conventional art. It is noted that, as used in the present application, the terms "automation," "automated", and "automating" refer to systems and processes that do not require human intervention. Although, in some implementations, a human editor or annotator may evaluate or otherwise review the CoM tracking performed by the devices and according to the processes described herein, that human involvement is optional. Thus, the CoM tracking of AV content provided by the devices and methods described in the present application may be performed under the control of the hardware processing components executing them.

<FIG> shows diagram <NUM> of an exemplary use case in which AV content <NUM> is manipulated by users 128b and 128c, according to one implementation. Included in <FIG> are AV content provider <NUM>, user 128a utilizing device 110a, user 128b utilizing device 110b, user 128c utilizing device 110c, and user 128d utilizing device 110d. Also shown in <FIG> are examples of modified AV content <NUM> and further modified AV content <NUM>.

According to one use case shown by <FIG>, user 128b may receive AV content <NUM> from AV content provider <NUM>. AV content provider <NUM> may be an owner or an owner authorized distributor of AV content <NUM> in the form of a video (including audio) of a movie, episodic television (TV) content that may include a web series and/or a video log, sports content, news content, or video gaming content, to name a few examples. According to an alternative use case also shown by <FIG>, user 128b may receive AV content <NUM> from device 110a controlled by user 128a. In that alternative use case, user 128a may utilize an AV content recording device integrated with device 110a, such as a video camera or cameras of device 110a, to generate AV content <NUM>. When generated by user 128a through use of device 110a, AV content <NUM> may be an amateur or professional artistic video, or a video of a live event, for example.

It is noted that the depictions of device 110a as a tablet computer, device 110b as a smartphone, device 110c as a laptop computer, and device 110d as a wearable personal communication device are merely exemplary. In various other implementations, one or more of devices 110a, 110b, 110c, and 110d (hereinafter "device(s) 110a-110d") may take the form of a video camera, a desktop computer, a laptop computer, a tablet computer, a gaming console, a smartphone, a smart television (smart TV), or a wearable personal communication device. Moreover, although wearable personal communication device 110d is shown as a headset, such as an augmented reality (AR) or virtual reality (VR) headset or glasses (hereinafter "AR/VR headset or glasses"), that representation is also merely exemplary. In other implementations, any of device(s) 110a-110d may be implemented as a wearable personal communication device in the form of a smartwatch, smart ring, or other smart personal item worn by one or more of users 128a, 128b, 128c, or 128d (hereinafter "users 128a-128d"), or situated in or on clothing worn by one or more of users 128a-128d.

As shown in <FIG>, user 128b utilizes device 110b to receive AV content <NUM> from AV content provider <NUM> or device 110a controlled by user 128a, and may use device 110b to manipulate AV content <NUM> to produce modified AV content <NUM>. As further shown in <FIG>, user 128c utilizes device 110c to receive modified AV content <NUM> from device 110b, and may use device 110c to further manipulate modified AV content <NUM> to produce further modified AV content <NUM>. User 128d utilizes device 110d to receive modified AV content <NUM> (produced by user 128b) from device 110c, or further modified AV content <NUM> (if produced by user 128c) from device 110c.

Manipulation of AV content <NUM> and modified AV content <NUM> is performed through use of respective AV processing applications, such as video recording or editing applications, for example, stored on devices 110b and 110c. The manipulation of AV content <NUM> or further manipulation of modified AV content <NUM> includes modification of one or more segments of audio content and/or video content included in AV content <NUM> and/or modified AV content <NUM>.

Where such a manipulation modifies video content of AV content <NUM> or modified AV content <NUM>, the manipulation may include one or more of changing the contrast of a video frame, deleting a video frame, inserting a video frame, removing an object from a video frame, inserting an object into a video frame, or changing colors within a video frame, to name a few examples. Where manipulation of AV content <NUM> or modified AV content <NUM> modifies audio content of AV content <NUM> or modified AV content <NUM>, such manipulation may include the deletion of a portion of original audio content and/or the insertion of additional audio content, such as music or speech, for example. Alternatively, or in addition, manipulation of audio content of AV content <NUM> or modified AV content <NUM> may include mixing audio tracks, changing the audio level of an audio track, or adding metadata to an audio track.

In the absence of a robust solution for tracking the manipulation of AV content <NUM> or further manipulation of modified AV content <NUM>, the owner of AV content <NUM>, e.g., AV content provider <NUM> or user 128a, cannot be assured that the AV content being distributed amongst downstream users 128b-128d is original content or an authorized copy or modification of original content. However, according to implementations of the present application, as described in greater detail below, device(s) 110a-110d are configured for use in a system for tracking the CoM of AV content <NUM>, which includes detecting and tagging the manipulations resulting in modified AV content <NUM>, and may include detecting and tagging the original creation of AV content <NUM> through use of device 110a.

In some implementations, tags enabling the authentication of original AV content <NUM> may be inserted in each segment of the AV content. Such authentication tags may include a facility identification tag identifying a production facility at which AV content <NUM> was created or modified AV content <NUM> was modified, for example. Other examples of authentication tags may include the Global Positioning System (GPS) location at which AV content <NUM> was created or modified AV content <NUM> was modified, a camera type, camera model, and lens parameters used in such a process, as well as the exposure time used to create or modify the content, to name a few.

<FIG> shows a diagram of exemplary device <NUM> for use in a system for performing automated CoM tracking of AV content <NUM>, according to one implementation. As shown in <FIG>, device <NUM> includes hardware processor <NUM>, AV content recording device <NUM>, system memory <NUM> implemented as a non-transitory storage device, and display <NUM>. According to the exemplary implementation shown in <FIG>, system memory <NUM> stores CoM tagging software code <NUM>, AV content processing application <NUM>, and optional content library <NUM>.

As shown in <FIG>, device <NUM> is utilized by user <NUM> within use environment <NUM> including communication network <NUM>, AV content provider <NUM>, and optional CoM history database <NUM>. It is noted that, in various implementations, optional CoM history database may take the form of a public or private secure ledger. Examples of such secure ledgers may include Blockchain, Hashgraph, Directed Acyclic Graph (DAG), and Holochain ledgers, to name a few. Also shown in <FIG> are one or more CoM tags <NUM> (hereinafter "CoM tag(s) <NUM>"), network communication links <NUM> communicatively coupling device <NUM> with AV content provider <NUM> and optional CoM history database <NUM>, and AV content <NUM>.

AV content provider <NUM> and AV content <NUM> correspond respectively to AV content provider <NUM> and AV content <NUM>, in <FIG>. As a result, AV content provider <NUM> and AV content <NUM> may share any of the characteristics attributed to respective AV content provider <NUM> and AV content <NUM> by the present disclosure, and vice versa. In other words, like AV content provider <NUM>, AV content provider <NUM> may be an owner or an owner authorized distributor of AV content <NUM>. Moreover, like AV content <NUM>, AV content <NUM> may take the form of a video and audio clip of a movie, episodic TV content that may include a web series and/or a video log, sports content, news content, or video gaming content delivered by AV content provider <NUM>, or may take the form of amateur or professional artistic video, or a video of a live event, for example, created by user <NUM> using device <NUM>.

Device <NUM> and user <NUM> correspond respectively in general to device(s) 110a-110d and users 128a-128d, respectively, in <FIG>. That is to say, device(s) 110a-110d may share any of the characteristics attributed to corresponding device <NUM> by the present disclosure, and vice versa. Thus, although not shown in <FIG>, device(s) 110a-110d may include features corresponding respectively to hardware processor <NUM>, AV content recording device <NUM>, display <NUM>, and system memory <NUM> storing CoM tagging software code <NUM>, AV processing application <NUM>, and optional content library <NUM>. In addition, although depicted as a smartphone in <FIG>, like device(s) 110a-110d, device <NUM> may take the form of any of a video camera, a desktop computer, a laptop computer, a tablet computer, a gaming console, a smartphone, a smart TV, or a wearable personal communication device such as a smartwatch or AR/VR headset or glasses, for example.

It is noted that although the present application refers to CoM tagging software code <NUM>, AV content processing application <NUM>, and optional content library <NUM> as being stored in system memory <NUM> for conceptual clarity, more generally, system memory <NUM> may take the form of any computer-readable non-transitory storage medium. The expression "computer-readable non-transitory storage medium," as used in the present application, refers to any medium, excluding a carrier wave or other transitory signal that provides instructions to a hardware processor of a computing platform, such as hardware processor <NUM> of device <NUM>. Thus, a computer-readable non-transitory medium may correspond to various types of media, such as volatile media and non-volatile media, for example. Volatile media may include dynamic memory, such as dynamic random access memory (dynamic RAM), while non-volatile memory may include optical, magnetic, or electrostatic storage devices. Common forms of computer-readable non-transitory media include, for example, optical discs, RAM, programmable read-only memory (PROM), erasable PROM (EPROM), and FLASH memory.

Communication network <NUM> may take the form of a packet-switched network such as the Internet, for example. Alternatively, communication network <NUM> may correspond to a wide area network (WAN), a local area network (LAN), or be implemented as another type of private or limited distribution network. As noted above, device <NUM> may be implemented as any of a video camera, a desktop computer, a laptop computer, a tablet computer, a gaming console, a smartphone, a smart TV, or a wearable personal communication device such as a smartwatch or AR/VR headset or glasses, for example. More generally, device <NUM> may be any suitable mobile or stationary computing devices that implement data processing capabilities sufficient to perform the functionality ascribed to device <NUM> herein. Furthermore, display <NUM> of device <NUM> may be implemented as a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, or any other suitable display screen that performs a physical transformation of signals to light.

<FIG> shows an exemplary diagram of modified AV content <NUM> including one or more CoM tags <NUM> (hereinafter "CoM tag(s) <NUM>") inserted by device 110b, according to one implementation, while <FIG> shows modified AV content <NUM> according to another implementation. Modified AV content <NUM> corresponds in general to modified AV content <NUM> in <FIG>. As a result, modified AV content <NUM> may share any of the characteristics attributed to corresponding modified AV content <NUM> by the present disclosure, and vice versa. Moreover, CoM tag(s) <NUM>, in <FIG>, correspond in general to CoM tag(s) <NUM>, in <FIG>, and those features can share any of the characteristics attributed to either corresponding feature by the present disclosure (described in further detail below with reference to <FIG>).

As shown in <FIG>, modified AV content <NUM> includes timecode <NUM>, and segments 332a and 332b each including a sequence of video frames. Thus, segment 332a of modified AV content <NUM> may include video frames 334a, 334b, 334c, and 334d (hereinafter "frame sequence 334a-334d"), while segment 332b of modified AV content <NUM> may include video frames 334e, 334f, <NUM>, and <NUM> (hereinafter "frame sequence 334e-<NUM>"). It is noted that although <FIG> depicts each of segments 332a and 332b as including four video frames, that representation is merely exemplary. In other implementations, the number of video frames in segments 332a and 332b may differ, and may number as few as one, or more than the four frames shown in Figure 1A.

As further shown in <FIG>, in some implementations, when manipulation of original AV content results in modification of one or more video frames of frame sequence 334a-334d and/or frame sequence 334e-<NUM>, one or more CoM tag(s) <NUM> track modification of the one or more video frames, and is/are inserted in a single frame of respective segment 332a and/or segment 332b. According to the exemplary implementation shown in <FIG>, for example, modification of any video frame of frame sequence 334a-334d results in CoM tag(s) <NUM> only being inserted into the first video frame of segment 332a (i.e., video frame 334a). Analogously, modification of any video frame of frame sequence 334e-<NUM> results in CoM tag(s) <NUM> only being inserted into the first video frame of segment 332b (i.e., video frame 334e).

It is noted that insertion of CoM tag(s) <NUM> into video frame 334a and 334e is merely exemplary, in other implementations, CoM tag(s) <NUM> may be inserted into one of video frames 334b, 334c, or 334d of segment 332a, and/or one of video frames 334f, <NUM>, or <NUM> of segment 332b, instead of being inserted into video frame 334a and/or video frame <NUM>, respectively. Thus, in implementations in which segments 332a and 332b include a sequence of video frames, CoM tag(s) <NUM> may be inserted into any video frame of the sequence of video frames such as a first video frame, a second or subsequent video frame, the last video frame, the video frame that has been modified, etc. Moreover, in use cases where no frame of a particular segment is modified, CoM tag(s) <NUM> would not be inserted into that segment. For example, if no frame of frame sequence 334e-<NUM> is modified, segment 332b would not be tagged by CoM tag(s) <NUM>.

In some implementations in which segments 332a and 332b include a sequence of video frames, CoM tag(s) <NUM> may be selectively inserted into a "key frame" or I-frame (hereinafter "I-frame") of the sequence of video frames. As known in the art, an I-frame refers to an "intra-coded picture" that includes all the information necessary to specify its content. By contrast, an accompanying "predicted picture" P-frame includes only differences from the previous frame, while a "bidirectional predicted picture" B-frame includes only differences from both the previous and subsequent frames.

By contrast to <FIG>, according to the implementation shown in <FIG>, each segment of modified AV content <NUM> includes only a single frame of video. Thus, segment 332a includes single video frame 334a, segment 332b includes single video frame 334b, and so forth. Thus, according to the implementation shown in <FIG>, each of video frames 334a, 334b, 334c, 334d, 334e, 334f, <NUM>, and <NUM> (hereinafter "video frames 334a-<NUM>") is a separate content segment. Moreover, according to the exemplary implementation shown in <FIG>, only video frames 334a, 334c, 334d, and <NUM> have been modified as a result of manipulation of original AV content. Therefore, only video frames 334a, 334c, 334d, and <NUM> have CoM tag(s) <NUM> inserted therein according to the implementation shown in <FIG>.

It is reiterated that manipulation of original AV content (e.g., AV content <NUM>/<NUM>) by results in the creation of modified AV content (e.g., modified AV content <NUM>/<NUM>). Like modified AV content <NUM>, original AV content (e.g., AV content <NUM>/<NUM>) may include timecode <NUM> and video frames 334a-<NUM> segregated into multiple segments corresponding respectively to segments 332a and 332b. Moreover, each of those segments may include sequences of video frames, or only a single video frame. It is further noted that the representation shown in <FIG> show modification of video content but not audio content included in original AV content for conceptual clarity. However, original AV content may include both audio content and video content, and modifications to audio content included in original AV content may be tracked using CoM tag(s) in a manner analogous to that shown in <FIG> for video content.

<FIG> shows an exemplary representation of one or more CoM tags <NUM> (hereinafter "CoM tag(s) <NUM>") suitable for use by device(s) 110a-110d/<NUM>, according to one implementation. As shown in <FIG>, CoM tag(s) <NUM> includes one or more of device identification tag 440a, and may include application tag 440b, user ID tag 440c, manipulation date tag 440d, timecode tag 440e, location tag 440f, checksums tag <NUM>, origination tag <NUM>, and metadata tag 440i. Moreover, and as further shown in <FIG>, in some implementations, CoM tag(s) <NUM> may further include CoM history database universal resource locator (URI) tag 440j including AV content ID <NUM>.

Device identification tag 440a identifies device 110b used for modifying AV content <NUM>/<NUM>. Application tag 440b identifies AV processing application <NUM> used for modifying AV content <NUM>/<NUM>. User ID tag 440c identifies a user performing the manipulation of AV content <NUM>/<NUM>, while manipulation date tag 440d is the date when the manipulation of AV content <NUM>/<NUM> occurs. Timecode tag 440e identifies the position or positions on timecode <NUM> at which modifications to AV content <NUM>/<NUM> is made, while location tag 440f may include GPS or other data identifying a geographical location of device 110b when used for modifying AV content <NUM>/<NUM>.

Origination tag <NUM> may include one or more tags identifying an original author of AV content <NUM>/<NUM>, an original creating device of AV content <NUM>/<NUM>, one or more applications used to create AV content <NUM>/<NUM>, or a creation data of AV content <NUM>/<NUM>. Metadata tag 440i may be metadata included in AV content <NUM>/<NUM> or may be metadata further describing distinctions between modified AV content and original AV content. Checksums tag <NUM> may include internal security and/or verification data included in AV content <NUM>/<NUM>. For example, a numerical value may be associated with each of CoM tag(s) <NUM>, and those values may be summed to generate checksums tag <NUM>. Any subsequent unauthorized modification to a segment of AV content <NUM>/<NUM> would cause the checksum for that segment to change relative to checksums tag <NUM> for that segment. In one implementation, checksums tag <NUM> may be a sum of values of one or more CoM tag(s) <NUM> of a segment, pixel values of the segment, or values of any other data or information associated with the segment.

CoM tag(s) <NUM> correspond in general to CoM tag(s) <NUM>/<NUM> in <FIG>, <FIG>. Consequently, CoM tag(s) <NUM>/<NUM> may share any of the characteristics attributed to corresponding CoM tag(s) <NUM> by the present disclosure, and vice versa. That is to say, although not shown in <FIG>, <FIG>, like CoM tag(s) <NUM>, CoM tag(s) <NUM>/<NUM> may include one or more of device identification tag 440a, application tag 440b, user ID tag 440c, manipulation date tag 440d, timecode tag 440e, location tag 440f, checksums tag <NUM>, origination tag <NUM>, metadata tag 440i, and CoM history database URI tag 440j including AV content ID <NUM>.

The functionality of device(s) 110a-110d/<NUM> including CoM tagging software code <NUM> will be further described by reference to <FIG>, in combination with <FIG>, <FIG>, <FIG>, and <FIG>. <FIG> shows flowchart <NUM> presenting an exemplary method for use by a device for performing automated CoM tracking of AV content, according to one implementation. With respect to the method outlined in <FIG>, it is noted that certain details and features have been left out of flowchart <NUM> in order not to obscure the discussion of the inventive features in the present application.

Referring now to <FIG> in combination with <FIG> and <FIG>, flowchart <NUM> begins with detecting manipulation of AV content <NUM>/<NUM> by AV processing application <NUM> modifying AV content <NUM>, wherein the manipulation of AV content <NUM>/<NUM> modifies at least one of video content or audio content of AV content <NUM>/<NUM> (action <NUM>). As noted above, in some implementations AV content <NUM>/<NUM> may be received by device 110b/<NUM> from AV content provider <NUM>/<NUM>, which may be an owner entity or an owner authorized distributor of AV content <NUM>/<NUM>. Alternatively, AV content <NUM>/<NUM> may be received by device(s) 110b110d/<NUM> from another user, such as another one or more of users 128a-128c/<NUM>. For example, AV content <NUM>/<NUM> may be received by device 110b/<NUM> from AV content provider <NUM>/<NUM> or by device(s) 110b-110d/<NUM> from another of user(s) 128a-128c/<NUM> via communication network <NUM> and network communication links <NUM>. As yet another alternative, AV content <NUM>/<NUM> may be stored locally on device(s) 110a-110d/<NUM>, and may be retrieved from optional content library <NUM>.

As further noted above, user 128b may use device 110b/<NUM> to manipulate AV content <NUM>/<NUM> to produce modified AV content <NUM>/<NUM>. The manipulation of AV content <NUM>/<NUM> may include modification of one or more segments 332a/332b of content included in AV content <NUM>/<NUM>. Where such a manipulation modifies video content of AV content <NUM>/<NUM>, the manipulation may include one or more of changing the contrast of one or more of video frames 334a-<NUM>, deleting one or more of video frames 334a-<NUM>, inserting one or more of video frames 334a-<NUM>, removing an object from one or more of video frames 334a-<NUM>, inserting an object into one or more of video frames 334a-<NUM>, or changing colors within one or more of video frames 334a-<NUM>, to name a few examples. Where such a manipulation modifies audio content of AV content <NUM>/<NUM>, the manipulation may include the deletion of a portion of original audio content and/or the insertion of additional audio content, such as music or speech, for example. Alternatively, or in addition, manipulation of audio content of AV content <NUM>/<NUM> may include mixing audio tracks, changing the audio level of an audio track, or adding metadata to an audio track.

Alternatively, and as also noted above, in some implementations, user(s) 128a-128d/<NUM> may utilize AV content recording device <NUM> integrated with respective device(s) 110a-110d/<NUM>, such as a video camera or cameras for example, to create or manipulate AV content <NUM>/<NUM>. For example, and as shown in <FIG> and <FIG>, user 128a/<NUM> may utilize AV content recording device <NUM> integrated with device 110a/<NUM> to create AV content <NUM>/<NUM>, while user 128b/<NUM> may utilize AV content recording device <NUM> integrated with device 110b/<NUM> to modify AV content <NUM>/<NUM>.

The creation or manipulation of AV content <NUM>/<NUM> described above may be performed through use of AV processing application <NUM>, which may be a video recording or editing application, for example, stored in system memory <NUM>. Detection of that manipulation, in action <NUM>, may be performed by CoM tagging software code <NUM>, executed by hardware processor <NUM>. For example, as shown in <FIG>, CoM tagging software code <NUM> is interactively coupled to AV content processing application <NUM>. As a result, use of AV content processing application <NUM> by device 110a/<NUM> to create original AV content <NUM>/<NUM>, as well as use of AV content processing application <NUM> by device 110b/<NUM> to modify original AV content <NUM>/<NUM> may be detected through communications between AV content processing application <NUM> and CoM tagging software code <NUM>.

Continuing to refer to the specific example in which original AV content <NUM>/<NUM> is modified by device 110b/<NUM>, flowchart <NUM> continues with generating, when the manipulation by AV processing application <NUM> is detected, CoM tag(s) <NUM>/<NUM>/<NUM> including device identification tag 440a identifying device 110b/<NUM> used for modifying AV content <NUM>/<NUM> (action <NUM>). It is noted that CoM tag(s) include at least device identification tag 440a, but may include one or more additional tags or data. For example, and as discussed above, in addition to device identification tag 440a, CoM tag(s) <NUM>/<NUM>/<NUM> may further include one or more of application tag 440b, user ID tag 440c, manipulation date tag 440d, timecode tag 440e, location tag 440f, checksums tag <NUM>, origination tag <NUM>, metadata tag 440i, and CoM history database URI tag 440j including AV content ID <NUM>.

Generation of CoM tag(s) <NUM>/<NUM>/<NUM> including device identification tag <NUM> identifying device 110b/<NUM> used for modifying AV content <NUM>/<NUM>, in action <NUM>, may be performed by CoM tagging software code <NUM>, executed by hardware processor <NUM> of device 110b/<NUM> used to modify AV content <NUM>/<NUM>. For example, mobile communication devices such as smartphones typically have a unique device identification number distinct from the serial numbers assigned respectively to its hardware components. In one implementation, hardware processor <NUM> may execute CoM tagging software code <NUM> to read that device identification number and generate device identification tag <NUM> including that number.

Flowchart <NUM> can conclude with inserting, in each of modified segments 332a/332b of AV content <NUM>/<NUM> modified by AV processing application <NUM>, CoM tag(s) <NUM>/<NUM>/<NUM> generated in action <NUM> (action <NUM>). Action <NUM> may be performed by CoM tagging software code <NUM>, executed by hardware processor <NUM>. With respect to CoM tag(s) <NUM>/<NUM>/<NUM>, it is noted that those tag(s) is/are persistent tags that cannot be altered by user(s) 128a-128d/<NUM> after insertion into modified segments of AV content <NUM>/<NUM> modified by AV processing application <NUM>. It is further noted that CoM tag(s) <NUM>/<NUM>/<NUM> are human and machine imperceptible when modified AV content <NUM>/<NUM> is played back by a playback device of device(s) 110a-110d/<NUM> on display <NUM>.

It is noted that any modifications to AV content <NUM>/<NUM> by an unauthorized agent would be made using a content processing application not monitored by CoM tagging software code <NUM>. As a result, those unauthorized modifications would be untagged. However, any modification made to AV content <NUM>/<NUM> results in a change in the checksum of each modified segment. Consequently, in implementations in which checksums tag <NUM> is included among CoM tag(s) <NUM>/<NUM>/<NUM>, any modifications made to a segment of AV content <NUM>/<NUM> by an unauthorized agent results in the checksum for that segment failing to match checksum(s) tag <NUM>, thereby identifying the content as invalid.

In many implementations, it may be advantageous or desirable for CoM tag(s) <NUM>/<NUM>/<NUM> inserted into modified AV content <NUM>/<NUM> to include all CoM tracking data. That is to say, in those implementations modified AV content <NUM>/<NUM> carries all information tracking modifications made to AV content <NUM>/<NUM> in its original state. However, in some implementations, some or substantially all of that tracking information may be stored remotely from device(s) 110a-110d/<NUM>, on CoM history database <NUM>, which may be implemented as a CoM history secure or distributed ledger such as, but not limited to, one of a Blockchain, Hashgraph, DAG, or Holochain ledger, for example.

In implementations in which some or substantially all of the CoM tracking information carried by CoM tag(s) <NUM>/<NUM>/<NUM> is stored on CoM history database <NUM>, CoM tag(s) <NUM>/<NUM>/<NUM> other than device identification tag <NUM> and CoM history database URI tag <NUM> including AV content ID <NUM> may be transferred to CoM history database <NUM>, as shown in <FIG>.

Although not included in the exemplary outline provided by flowchart <NUM>, in some implementations, the present method may further include rendering AV content <NUM>/<NUM> created by device 110a210 and/or modified AV content <NUM>/<NUM> produced by device 110b/<NUM> on display <NUM>. As noted above, display <NUM> may be implemented as an LCD, an LED display, an OLED display, or any other suitable display screen that performs a physical transformation of signals to light. Rendering of AV content <NUM>/<NUM> created by device 110a/<NUM> and/or modified AV content <NUM>/<NUM> produced by device 110b/<NUM> on display <NUM> may be performed using AV content processing application <NUM>, executed by hardware processor <NUM>.

CoM tag(s) <NUM>/<NUM>/<NUM> inserted into AV content <NUM>/<NUM> created by device 110a210 and/or modified AV content <NUM>/<NUM> produced by device 110b/<NUM> may advantageously be used by AV content provider <NUM>/<NUM> to validate the authenticity of AV content <NUM>/<NUM> and/or modified AV content <NUM>/<NUM>. For example, AV content provider <NUM>/<NUM> or an authorized third party authenticator acting on behalf of AV content provider <NUM>/<NUM> may analyze the CoM history recorded by CoM tag(s) <NUM>/<NUM>/<NUM> to identify the entity that created or initiated distribution of AV content <NUM>/<NUM>, as well as any and all modifications made to AV content <NUM>/<NUM> and resulting in modified AV content <NUM>/<NUM>.

Thus, the present application discloses robust solutions for performing automated CoM tracking of AV content that overcome the drawbacks and deficiencies in the conventional art. From the above description it is manifest that various techniques can be used for implementing the concepts described in the present application without departing from the scope of the invention which is defined by the appended claims.

Moreover, while the concepts have been described with specific reference to certain implementations, a person of ordinary skill in the art would recognize that changes can be made in form and detail without departing from the scope of the invention which is defined by the appended claims.

Claim 1:
A method for tracking a chain-of-manipulation, CoM, of an audio-video, AV, content (<NUM>, <NUM>) having a plurality of segments (332a, 332b), the method comprising:
detecting, by a CoM tagging software code (<NUM>), a modification of the AV content (<NUM>, <NUM>) by an AV processing application (<NUM>) modifying the AV content (<NUM>, <NUM>), wherein the modification modifies at least one of a video content or an audio content of one or more segments (332a, 332b) of the AV content (<NUM>, <NUM>);
generating, by the CoM tagging software code (<NUM>) after detecting the modification of the AV content (<NUM>, <NUM>) by the AV processing application (<NUM>), one or more CoM tags (<NUM>, <NUM>) to track the modification, wherein the one or more CoM tags (<NUM>, <NUM>) include a device identification tag (440a) identifying the device (110b) which performed the modification of the AV content (<NUM>, <NUM>); and
inserting, by the CoM tagging software code (<NUM>), the one or more CoM tags (<NUM>, <NUM>) in each of the one or more modified segments (332a, 332b), wherein the inserted one or more CoM tags (<NUM>, <NUM>) are human and machine imperceptible when the AV content (<NUM>, <NUM>) is played back by a playback device on a display (<NUM>).