Patent Description:
The present disclosure generally relates to the use of data processing systems and methods for modifying a video stream. A video stream may generally be considered to be a sequence of digital signals, or packets of data, representing a plurality of frames of image data. Video streams may also comprise other data, such as audio data, and metadata which can be used to process the video stream to display a video to a user via a suitable media player.

Videos are captured in many different environments, and for a number of different purposes. Purely by way of example, a video may be captured for the purpose of entertainment, surveillance, and for analysing real-world environments. Video data, captured using a video capture device, can be transmitted from the video capture device to computing devices. Such computing devices may perform any number of functions including, for example, storing, compressing, encoding, transmitting and/or receiving the video data. Video data may be sent, or streamed, to a plurality of different such computing devices for receipt by a plurality of users.

In some cases, a particular sequence of video data may be modified differently to suit different recipients. It would be desirable to provide an improved process for transmitting the video data to these different recipients.

<CIT> discusses applying customized permission settings to protect particular regions of a document, and, in particular, documents that are of a 'lossy' encoded media file type, e.g., an MPEG video format.

<CIT> discuses generating redacted video of a scene captured using a video camera and augmenting a manual review process of the redacted video of the scene.

<CIT> discusses masking and restoring the image of an image system for protecting personal privacy.

<CIT> discusses privacy protection of personal images online and in particular within social media networks.

<CIT> discusses a masking and restoring method of a privacy domain, provided to minimize the amount of data which is stored or transmitted.

Aspects of the present invention are set out according to the appended claims.

The use of video recording equipment in public and private spaces is ubiquitous and the processing and distribution of the resulting video footage presents a number of technical challenges. One such challenge is to manage security and privacy while enabling the video footage to be used for its intended purpose.

For example, video surveillance footage, captured for security purposes, is often used by people with the relevant authorization, such as the police, to identify specific people, objects, and/or to provide evidence of events, such as crimes, which may occur in a given location. In this example, the video footage may also comprise sensitive information which is not relevant for the purposes described above. It is desirable that sensitive, or private, information is not distributed and/or shown to people without the relevant authorization to access this information. Surveillance footage, captured in a public space, may show the faces of people who were present in the public space as well as other sensitive information, such as pictures of identification cards, credit cards, private or confidential paperwork, and/or objects. Furthermore, what is considered sensitive, or private, information may depend on an environment in which the video is captured. In public spaces, little information captured in a video may be considered private. In private facilities, such as offices, factories, and homes, a relatively larger amount of information which is captured in a video may be considered sensitive or private.

While the example above relates to surveillance footage, video data, captured for other purposes aside from surveillance, may also contain a combination of sensitive and non-sensitive information. As the quality of video footage and recording equipment increases, the number of applications for which video footage is used is commensurately increasing. For example, recording video footage may be used as part of a manufacturing pipeline in a factory to monitor and/or control certain processes. The captured video footage may be processed automatically, using computer-vision techniques, or may be reviewed manually by human operators. In this case, products being manufactured, or parts of the manufacturing process itself, can be confidential. Other examples include monitoring office spaces, traffic, pedestrian flow, and even for environmental purposes to monitor movements and/or numbers of flora and/or fauna in a given area. In all of these cases, sensitive information may be captured either intentionally or unintentionally.

Digital videos are generally stored and transmitted in the form of video streams. Video streams may comprise one or more bitstreams such as an audio data bitstream and an image data bitstream. Video streams may be stored as files having suitable container formats, for example, MP4, AVI, and so on, which enable a single file to store multiple different bitstreams. Bitstreams may be arranged into a plurality of data packets, each of which contains an integer number of bytes. Data packets comprise a header portion, specifying (among other information) the type of payload, and the payload itself, which in the case of video may be Raw Byte Sequence Payload (RBSP). The types and number of data packets depend on the techniques which are used to generate the bitstreams. For example, the H. <NUM> Advanced Video Compression (H. <NUM> AVC) standard provides a bitstream comprising a sequence of Network Abstraction Layer (NAL) packets having a payload comprising N bytes. There can be <NUM> different types of packets which may be included in the bitstream. Examples of packet types in the H. <NUM> AVC standard include, undefined, slice data partition A, slice data partition B, slice data partition C, sequence parameter set, picture parameter set, and so forth.

An image data bitstream comprises a plurality of frames of image data and is generated by processing raw image data from an image sensor into a format which is suitable to be viewed using a media player application on a computer device. The plurality of frames of image data may be arranged sequentially in an image data bitstream and/or indications of the respective positions of each frame of image data may be included in the image data bitstream, e.g. timestamps, such that the frames of image data can be viewed in the correct order using a media player application.

A frame of image data comprises data representing a plurality of pixel intensity values generated by at least one image sensor. An image sensor is a sensor which converts light into a digital signal representing an intensity of light incident on the image sensor. Image sensors may operate in the visible light spectrum, but may additionally, or alternatively, include sensors which operate outside of the visible light spectrum, for example, the infrared spectrum. By using at least one image sensor to capture a plurality of frames of image data, for example, at least two frames of image data, sequentially in time, it is possible to generate a video. In other implementations, a video may be generated from at least two frames of image data which are not captured sequentially in time.

Video streams are generally encoded to compress the video stream for storage and/or transmission. Encoding involves using a video encoder implemented as any suitable combination of software and hardware, to compress a digital video. There is a plurality of techniques that can be used to encode videos. One such technique is specified in a technical standard commonly referred to as the H. <NUM> AVC standard, mentioned above. Video encoders, like those implementing the H. <NUM> AVC standard, encode videos based on macroblocks of pixels in the video stream. A macroblock is a unit used in image and video compression which comprises a plurality of pixels. For example, image data can be grouped into macroblocks of <NUM>×<NUM> pixels for the purposes of encoding.

Macroblocks are encoded by performing one or more transform operations, such as integer transform, to generate transform coefficients. Quantisation is performed on the resulting transform coefficients, and the quantised transform coefficients are subsequently entropy encoded to produce an encoded bitstream.

In some implementations, macroblocks may be processed in slices, a slice being a group of one or more macroblocks generally processed in raster scan order. Each frame of image data is generally made up of a plurality of slices, although in some cases a frame of image data may be constructed from a single slice. Some video encoding and compression standards provide support for so-called flexible macroblock ordering. This allows macroblocks to be grouped, processed, encoded, and sent in any direction and order, rather than just raster scan order. In particular, macroblocks can be used to create shaped and non-contiguous slice groups. Some specifications for video compression, for example the mentioned H. <NUM> AVC standard, specify different types of scan patterns of the macroblocks for generating slices including, for example, interleaved slice groups, scattered or dispersed slice groups, foreground groups, changing groups, and explicit groups.

Certain examples described herein relate to systems and methods for processing a video stream to modify image data comprised in the video stream. Portions of image data in the video stream may be selected and modified based on one or more predetermined criteria. Where there are security and/or privacy related considerations for the video stream, e.g. where the video stream is to be distributed to a large number of people, the modification of the selected portions may function to increase the security and/or privacy of relevant portions of the image data.

Certain examples, described below, involve modifying portions of image data by changing the portions of image data so that they omit information which was in the unmodified portion of image data. Other examples provide techniques in which modifying portions of image data includes encrypting the portions of image data to inhibit or prevent unauthorised access to the portions of image data.

<FIG> shows a data processing system <NUM> suitable for processing video streams that are the subject of the mentioned examples and embodiments. The system <NUM> comprises at least one processor <NUM> and storage <NUM>, the storage <NUM> comprising computer-executable instructions <NUM>. In some examples, the storage <NUM> may also comprise computer program code for implementing one or more further modules such as an object detection module <NUM> and a decoder <NUM>. The data processing system <NUM> may also comprise further elements, not shown in <FIG> for simplicity. For example, the data processing system <NUM> may further comprise one or more communication modules, for transmitting and receiving data, further software modules, for performing functions beyond those described here, and one or more user interfaces, to allow the data processing system <NUM> to be operated directly by a user.

The processor(s) <NUM> comprises any suitable combination of various processing units including a central processing unit (CPU), a graphics processing unit (GPU), an Image Signal Processor (ISP), a neural processing unit (NPU), and others. The at least one processor <NUM> may include other specialist processing units, such as application specific integrated circuits (ASICs), digital signal processors (DSPs), or field programmable gate arrays (FPGAs). For example, the processor(s) <NUM> may include a CPU and a GPU which are communicatively coupled over a bus. In other examples, the at least one processor <NUM> may comprise a CPU only.

The storage <NUM> is embodied as any suitable combination of non-volatile and/or volatile storage. For example, the storage <NUM> may include one or more solid-state drives (SSDs), along with non-volatile random-access memory (NVRAM), and/or volatile random-access memory (RAM), for example, static random-access memory (SRAM) and dynamic random-access memory (DRAM) Other types of memory can be included, such as removable storage synchronous DRAM, and so on. The computer-executable instructions <NUM> included on the storage <NUM>, when executed by the at last one processor <NUM>, cause the processor(s) <NUM> to perform a computer-implemented method for processing a video stream as described herein.

The data processing system <NUM> may be implemented as part of a single computing device, or across multiple computing devices as a distributed data processing system. The data processing system <NUM> may be included in a device used for capturing video (or a video camera). For example, the data processing system <NUM> can be included as part of the processing circuitry in the video camera. Alternatively, the data processing system <NUM> may be separate from the video camera and included in an image processing pipeline which includes the video camera. In this case, the data processing system <NUM> is connected to an output of the video camera over any suitable communication infrastructure, such as local or wide area networks, through the use of wired and/or wireless means.

A computer-implemented method <NUM> for processing a video stream according to an embodiment and which is implemented by the data processing system <NUM>, will now be described with reference to <FIG>. <FIG> shows a flow diagram illustrating the computer-implemented method <NUM>, and <FIG> show block diagrams of example implementations of the computer-implemented method <NUM>.

With reference to <FIG> and <FIG>, at a first block <NUM> of the flow diagram, the data processing system <NUM> receives a video stream <NUM> comprising a plurality of frames of image data 304a to <NUM>.

In some examples, a video camera used to generate the video stream <NUM> may include an encoder to encode image data generated by an image sensor in the video camera. In other implementations, the video camera may output a video stream <NUM> which has not been encoded. Accordingly, the video stream <NUM> which is received by the data processing system <NUM> may be encoded or unencoded. If the video stream <NUM> is received in an encoded format the data processing system <NUM> decodes the video stream <NUM> using suitable algorithms such as those provided by a decoder <NUM> which implements a decoding technique which corresponds to the encoding technique used to encode the video stream <NUM>.

At a second block <NUM>, the data processing system <NUM> selects portions of the image data on the basis of first data <NUM> indicating locations within the image data. The first data <NUM> may be implemented in a number of ways, for example, the first data <NUM> may include a copy of portions of the image data which are to be selected, pointers to the portions of the image data which are to be selected (e.g. metadata associated with data packets comprising portions of image data to be selected), one or more timestamps relating to a location in a bitstream comprising the image data, indications of macroblocks of image data which are to be selected. In the present example, the first data <NUM> comprises one or more image masks each associated with at least one frame of the plurality of frames of image data. The image masks each identify regions in at least one frame of image data which are to be selected.

The selected portions of the image data represent objects in the frames of image data which are to be modified, for example, to redact or obscure information in the image data. The image masks (the first data <NUM>), may be generated automatically using object detection algorithms performed on the image data. For example, image masks can be generated by processing the plurality of frames of the image data 304a to <NUM> using computer vision techniques to identify, (or detect), at least one predetermined class of object represented by the image data.

Where computer vision techniques are used to generate the image masks, the one or more predetermined classes may comprise any class of object which is detectable by computer-vision techniques. Examples of predetermined classes of objects include buildings, people, personal belongings, specific groups of people, specific individuals, and specific parts of people, such as their faces. Predetermined classes may also comprise sensitive documentation such as credit cards, identification cards, confidential documents, vehicle registration and/or licence plate numbers, and so on. One or more classifiers, such as an artificial neural network, can be used as part of a computer-vision technique to identify the at least one predetermined class of object.

Various combinations of artificial neural networks can be used to identify at least one predetermined class of object represented by the image data, for example a combination of two more convolutional neural networks comprising different architectures or trained to detect different classes of objects. Artificial neural networks used for object identification may include any suitable number of layers, including an input layer, one or more hidden layers, and an output layer. When using a combination of two or more neural networks, an output from a first artificial neural network may be provided to the input of one or more second artificial neural networks. Outputs from a plurality of artificial neural networks may be combined using suitable mathematical operations, as will be appreciated by a person skilled in the art.

Alternatively, or additionally, at least one of the image masks may be generated based on an input from a user. A user may view the video on a display and may use one or more user interfaces to select regions in the video which are to be modified. For example, a user may select the face of a person within the video which is to be redacted. An image mask may then be generated based on the selection of the face. In some examples, generating image masks may include tracking the face across multiple frames of image data and generating image masks which identify all relevant portions of image data, representing the face, which are to be modified.

The first data <NUM>, such as one or more image masks, may be generated locally at the data processing system <NUM>, such as where the data processing system <NUM> comprises an object detection module <NUM>. Alternatively, the first data <NUM> may be generated remotely from, and transmitted to, the data processing system <NUM>. In either case, the first data <NUM> may be stored in the storage <NUM> and the method <NUM> may comprise accessing the storage <NUM> to retrieve the first data <NUM>.

The data processing system <NUM> modifies, at a third block <NUM>, the selected portions of the image data to generate modified portions of the image data 308a to 308c. Modifying the selected portions of the image data may involve performing one or more operations on the selected portions of image data to obscure information represented by the selected portions. For example, where the selected portions of the image data represent sensitive or private information, such as pictures of faces, modifying the selected portions of the image data comprises obscuring the faces by performing suitable operations on the selected portions of image data. The resulting modified portions of the image data 308a to 308c may comprise a blurred, redacted, or otherwise censored version of the unmodified image, and the private or sensitive information may be replaced with a sign or symbol indicating that those parts of the image are sensitive and/or have been censored.

Modifying the selected portions of the image data may comprise modifying the selected portions of the image data to remove sensitive or private information while obscuring the fact that the selected portions have been modified. For example, modifying the selected portions may comprise replacing the selected portions of the image data with image data representing a background in the image, such that the sensitive or private information is removed without replacing the sensitive information with blur or noise. The image data representing the background may be generated based on previous frames of image data in which the sensitive or private information is not present at the particular location in the frame of image data, so that only the background is visible. Alternatively, or additionally, the image data representing a background in the image may be artificially generated based on the unselected portions of the image data 312a to 312e.

At a fourth block <NUM>, the data processing system <NUM> generates second data <NUM> for use in recovering at least some of the selected portions from the modified portions 308a to 308c. The second data <NUM> which is generated, and the manner in which it can be used to recover the selected portions, is dependent on the method used to modify the selected portions. In some examples, the second data <NUM> comprises data identifying the location of the modified portions of the image data 308a to 308c. The location of the modified portions of the image data 308a to 308c may be identified with respect to the unselected portions of the image data 312a to 312e and/or with respect to a video stream comprising the modified and unselected portions of the image data. To this end, the second data <NUM> may comprise metadata identifying the modified portions of the image data 308a to 308c, e.g. metadata associated with header portions of data packets comprising modified portions of the image data 308a to 308c.

In certain examples, such as those which will be described below in relation to <FIG>, the second data <NUM> may additionally, or alternatively, comprise differences between selected portions of the image data and respective modified portions of the image data 308a to 308c. Differences, in this context, may relate to differences between pixel intensity values represented by the modified portions of the image data 308a to 308c and pixel intensity values represented by respective selected portions of the image data. Generating second data <NUM> which can be used in recovering the selected portions of image data allows the original video stream <NUM> to be recovered from the modified portions 308a to 308c and the unselected portions of the image data 312a to 312e. As such, selected portions and hence the original video stream <NUM> can be discarded. This reduces storage requirements that are otherwise necessary to maintain both a modified version of the video stream, in which certain parts of the images are modified, and the original video stream <NUM>.

The data processing system <NUM> outputs, at a fifth block <NUM>, the second data <NUM>, for example, in the form of differences between the modified portions of the image data 308a to 308c and the selected portions of the image data, together with unselected portions of the image data 312a to 312e, and the modified portions of the image data 308a to 308c. This enables a modified version of the video stream <NUM> to be distributed, in the form of the modified portions 308a to 308c and the unselected portions of the image data 312a to 312e, to users to view the video. By distributing the modified portions of the image data 308a to 308c and the unselected portions of the image data 312a to 312e it is possible to provide users who wish to view the video stream <NUM> with a version in which certain information may be obscured, depending on the selection of the portions of the image data. In this way, security and/or privacy related objectives may be achieved while also providing relevant parts of the video stream <NUM> to users.

In some examples, outputting the modified portions of the image data 308a to 308c and the unselected portions of the image data 312a to 312e comprises generating an output video stream <NUM>. The output video stream <NUM> comprises a further plurality of frames of image data, where the further plurality of frames of image data includes the modified portions of image data 308a to 308c and the unselected portions of the image data 312a to 312e. In this way it is possible for the output video stream <NUM> to be stored in a file in a suitable container format to be viewed using an appropriate media player. This may include sending the file to users and/or providing a web-based media player allowing users to view the output video stream <NUM> remotely.

Conversely, when the further plurality of frames of image data do not include the selected portions of the image data, it is not possible for viewers of the output video stream <NUM> to view or reconstruct the original video stream <NUM> from the output video stream <NUM>. This maintains the privacy and/or security of the selected portions of image data.

For users who are authorised to view the original video stream <NUM> the second data <NUM>, e.g. in the form of differences between modified and selected portions of the image data, may be provided as an output data stream, so that these users may recover the selected portions of the image data and hence reconstruct the original video stream <NUM>. To this end, the second data <NUM> can be generated in a suitable format, such as a data packet, or an NAL packet according to the H. <NUM> AVC standard. To enable efficient transfer of the second data <NUM>, encoding methods can be applied to a data packet comprising the second data <NUM>. The second data <NUM> may also be embedded in the output video stream <NUM>. This allows the second data <NUM> to be provided to a recipient who is already set up to receive the output video stream <NUM>. Where the modified portions of the image data 308a to 308c and the unselected portions of the image data 312a to 312e are included in an output video stream <NUM>, the output video stream <NUM> may be encoded using any suitable encoding method. In some examples, such as those described below in relation to <FIG>, encoding the second data <NUM> and/or the further plurality of frames of image data in the output video stream <NUM> may involve use of e.g. standards complaint encoding methods. In other examples, such as those described in relation to <FIG>, the encoding may involve use of modified standards compliant encoding methods.

In the example shown in <FIG>, the encoding processes, used to encode the second data <NUM> and the output video stream <NUM>, include standards compliant encoding techniques, e.g. encoding techniques implemented by a general H. <NUM> encoder. The video stream <NUM> is received at the data processing system <NUM> and input into first module <NUM>, for selecting and modifying portions of image data, and a difference engine <NUM>.

The first module <NUM> receives the video stream <NUM> and the first data <NUM>. The first module <NUM> uses the first data <NUM> to select portions of the image data included in the video stream <NUM>. As described above, the first data <NUM> may include one or more image masks identifying locations within the image data. For example, the image masks may identify parts of the image data (or regions of the images) representing private or sensitive information. Selected portions of the image data may include sub-frame sized portions of the image data and/or whole frames of image data.

The portions of the image data which are selected are then modified to conceal information represented by the selected portions of the image data. Modifying the selected portions of the image data may involve the use of one or more operations, e.g. applying one or more filters to the selected portions of the image data, replacing the selected portions of the image data with randomly or pseudo-randomly generated image data, and/or replacing the selected portions with alternative predetermined image data. The first module <NUM> produces the output video stream <NUM> which includes a further plurality of frames of image data. The further plurality of frames of image data may be arranged in the same sequence as the plurality of frames of image data 304a to <NUM> in the video stream <NUM>, but with the selected portions of the image data being replaced in line with the modified portions of the image data 308a to 308c.

The output video steam <NUM>, output from the first module <NUM>, is provided to the difference engine <NUM>. The difference engine <NUM> compares the plurality of frames of image data 304a to <NUM>, included in the video stream <NUM>, with respective frames of image data in the further plurality of frames of image data from the output video stream <NUM> to generate difference data <NUM>. The difference data <NUM> comprises differences between the selected portions of the image data and respective modified portions of the image data 308a to 308c. For example, for each frame of image data 304a to <NUM>, the difference data <NUM> may comprise one or more image masks. The image masks specify the differences between the selected portions of image data and the modified portions of image data 308a to 308c. The image masks may also identify the positions of the modified portions of the image data 308a to 308c with respect to the unselected portions of image data in the output video stream <NUM>.

The difference data <NUM> is processed to generate second data <NUM> which is encrypted at a second module <NUM>. Accordingly, the second data <NUM> includes at least some of the difference data <NUM> and hence a difference between at least one of the selected portions of the image data and a respective modified portion of the image data 308a to 308c. The modified portion of the image data 308a to 308c can be processed based on a difference represented by the second data <NUM> to recover the selected portion of the image data. Encrypting the differences between the modified and selected portions may involve using any suitable encryption method to secure the differences and inhibit unauthorised persons from accessing the differences. By encrypting the second data <NUM>, including the differences between the selected and modified 308a to 308c portions of the image data, it is possible to embed the second data <NUM> in the output video stream <NUM> which can be widely distributed while ensuring that only authorised persons are able access the difference information for recovering the selected portions of the image data. Hence a single output video stream <NUM> may be provided by which different users have access to different versions of the video represented by the video stream <NUM>. For example, certain users may be provided with an encryption key to decrypt and access the differences between the selected and modified 308a to 308c portions of the image data, while other users are not provided access to encryption keys.

Encrypting the second data <NUM> may, in some instances, include encrypting the second data <NUM> according to a single-use encryption key. A single-use encryption key allows the second data <NUM> to be decrypted once so that it can be used to recover the selected portions. Thereafter, the second data <NUM> is no longer able to be decrypted by a user to whom it is provided. This allows the selected portions of the image data to be recovered for a single limited and specific circumstance while maintaining the general security of the information represented by the second data <NUM>. This is of particular use where the method <NUM> is implemented for a video stream <NUM> which contains highly sensitive information. In one such example, the video stream <NUM> represents surveillance footage which is relevant to a criminal investigation. In this case, it is desirable to temporarily recover the selected portions of the image data such that the unredacted video stream <NUM> can be viewed by a jury. After the unredacted video stream <NUM> has been viewed by the jury it is desirable that the second data <NUM> cannot be decrypted again so that any unauthorized persons, who obtain the second data, are not able to recover the selected portions of the image data and thereby can only view the redacted video stream <NUM>. Where a single-use encryption key is used to encrypt at least part of the second data <NUM>, the single-use encryption key may be a pre-shared encryption key which is provided to a target recipient of the second data <NUM> through a secure communications channel.

The second data <NUM> may additionally include metadata, for example metadata relating to the encryption methods used to encrypt the second data <NUM>, metadata for use in identifying a relative position of the modified portions of the image data 308a to 308c in the output video stream <NUM>, and other metadata suitable for use combining the recovered selected portions of the image data and the unselected portions of the image data 312a to 312e to generate a video.

The metadata included in the second data <NUM> may also include an indication that the second data <NUM> is available for a single use to recover at least some of the selected portions of the image data from the modified portions of the image data 308a to 308c. In other words, the second data <NUM> may include metadata specifying that the second data <NUM> is limited to a single use. To prevent the second data <NUM> from being used more than once to recover the selected portions of the image data, the metadata may include an instruction to modify the second data <NUM> after it has been used to recover selected portions of the image data from the modified portions of the image data 308a to 308c. Modifying the second data <NUM> in this case may include corrupting and/or deleting the second data <NUM>.

Where the output video stream <NUM> and/or the second data <NUM> are encoded by the data processing system <NUM>, encoding blocks <NUM> and <NUM> are used to encode the respective outputs. In this example, the encoding blocks <NUM> and <NUM> may involve use of e.g. standards compliant encoding methods, such as an implementation of encoding methods specified by the H264 AVC standard mentioned above.

In contrast to the example shown in <FIG>, in which the video stream is processed at the frame level, in the example shown in <FIG>, portions of the image data are selected at the macroblock level. In other words, the selected portions of the image data comprise an integer number of macroblocks. In this case, the first data <NUM> may indicate a location within the video stream <NUM> and identify an integer number of macroblocks which are to be selected. Alternatively, or additionally, the first data <NUM> may identify each macroblock which is to be selected. The macroblocks which are to be selected may be located sequentially within the video stream <NUM>. In other implementations, the first data <NUM> may indicate the locations of a plurality of non-sequential groups of macroblocks, wherein each group of macroblocks comprises one or more macroblocks arranged contiguously within the video stream <NUM>. To identify macroblocks of the image data, first data <NUM> may include metadata associated with one or more data packets comprising respective payloads each representing at least part of a macroblock of image data which is to be selected.

In certain encoding standards, such as are specified in the H. <NUM> AVC standard, encoding image data for a macroblock involves using predictions generated based on neighbouring macroblocks. These predictions may be intraframe (generated from information within a single frame of image data) or interframe (generated using information from macroblocks in previous and/or subsequent frames of image data). Residuals, generated from differences between input values, included in a macroblock, and predicted values, included in macroblock predictions which are generated based on neighbouring blocks, are encoded to generate an output video stream.

Where portions of the image data which are selected <NUM> and modified <NUM> are macroblocks, artefacts can be introduced which are not contemplated in general standards-compliant encoding techniques. If such artefacts are processed using standards-compliant encoding techniques, errors will be introduced. Therefore, in order to ensure these artefacts are properly processed, a customised technique (which may or may not be based on the standards-compliant techniques) is preferably used in the predictions for a macroblock. In the present example, illustrated in <FIG>, changes are made to general standards-compliant processing and encoding pipelines to mitigate errors which may otherwise occur when selecting and modifying macroblocks of image data. A macroblock <NUM> from the video stream <NUM> is selected and modified, at block <NUM>, based on the first data <NUM>. At difference engine <NUM>, a prediction for the macroblock is subtracted from the modified macroblock of image data. Generating the prediction follows a process which is similar to standards-compliant techniques as specified in the H. <NUM> AVC standard but with some modifications as will become apparent from the following description. This process includes inversely transforming <NUM> the quantised transform coefficients output from part of the encoding pipeline. The process for generating predictions further includes using a deblocking filter <NUM>, generating intra-frame predictions <NUM>, and making inter/intra frame predication decisions <NUM> based on the output of the prediction <NUM>, and motion compensation <NUM>. The motion compensation <NUM> is generated from the output of a buffer <NUM> which is attached to an output of the deblocking filter <NUM>, and an output from a motion estimation module <NUM>. The modification made to standards-compliant techniques here is that an output from the selection and modification block <NUM> is used in the motion estimation block <NUM> to compensate for errors which may arise due to the modification of the macroblock <NUM>. For example, an indication that a specific macroblock has been modified may allow the motion estimation block <NUM> to exclude the modified macroblock from calculations involved in estimating motion.

The macroblock prediction is used to encode the modified macroblock, by generating residuals at difference engine <NUM>, and to encode an unmodified version of the macroblock, by generating residuals at difference engine <NUM>. Both the residuals for the modified and the unmodified version of the macroblock are encoded. The encoded modified macroblock is output at block <NUM>. The residuals for the unmodified version of the macroblock are used to generate second data <NUM>, which is encoded and encrypted, at block <NUM>. An encrypted data packet <NUM>, comprising the second data <NUM> and which can be provided to users using a data stream, is output from the data processing system <NUM>.

In other examples, the selection and modification of portions of the image data may be performed at the slice level. In other words, the portions of image data which are selected may comprise one or more macroblocks arranged in any suitable slices. Where flexible macroblock ordering is available, the sizes of slices may vary.

In some examples, encrypting the second data <NUM> comprises encrypting a first part of the second data <NUM> according to a first security trait and encrypting a second part of the second data <NUM> according to a second security trait. Encrypting the first and second parts of the second data <NUM> according to different security traits allows more granularity in the selection of which users are capable of viewing unselected portions of the image data by decrypting the second data <NUM>. The first and second parts of the second data <NUM> may be different parts of the second data <NUM>, for example, at least one of the first and second parts may comprise data not included in the other of the first and second parts of the second data <NUM>. Where the second data <NUM> includes differences between selected portions of the image data and modified portions of the image data 308a to 308c, the first part of the second data <NUM> may include differences between a first set of some of the modified portions of the image data 308a and a respective first set of the selected portions of the image data, while the second part of the second data <NUM> may include differences between a second set of some of the modified portions of the image data 306b to 306c and a respective second set of the selected portions of the image data. In this way, it may be possible to allow different users to each view a different version of a video, for example, each version having at least one different recovered selected portion than the other versions of the video. It will be appreciated that some parts of the second data <NUM>, for example including metadata for use in identifying positions of the modified portions of image data 308a to 308c in the output video stream <NUM>, may not be encrypted.

Each of the first and second security traits may be associated with a respective different set of one or more encryption keys. By encrypting first and second sets of differences in the second data <NUM> according to different security traits, it is possible to enable different encryption keys to be used to decrypt the first and second sets of differences. For example, a first user may have access to encryption keys which are suitable for decrypting both the first and second sets of differences and hence can recover all of the selected portions of the image data. A second user may have access to an encryption key which is suitable for decrypting only the first set of differences of the second data <NUM>, and hence is able to recover only the selected portions of the image data associated with the first set of differences. Other users may have access to no encryption keys, or encryption keys suitable for decrypting only the second set of differences in the second data <NUM>.

By encrypting different parts of the second data <NUM>, for examples sets of differences between selected and modified portions of the image data, according to respective security traits a single output from the data processing system <NUM>, e.g. the second data <NUM>, can be provided to a plurality of users, while managing and controlling which of the plurality of users can recover the selected portions of the image data. This provides an efficient way of securely distributing video to a plurality of users having different authorization levels.

Access to the second data <NUM> may also be controlled based on a predetermined time interval, such that the second data <NUM> can only be used to recover the selected portions of the image data within a predetermined time period. Limiting access to the second data <NUM> to recover the selected portions of the image data based on a predetermined time period provides greater flexibility when controlling access to the unredacted video stream <NUM>. This is implemented by providing the second data <NUM> with metadata comprising an expiry date value, indicating a date and/or time at which access to the second data <NUM> is to expire. The expiry date value may be evaluated when attempting to process the modified portions of the image data 308a to 308c using the second data <NUM> to determine whether the second data <NUM> is valid and allowed to be used in this way.

In other examples, the second data <NUM> is made available to a user only within the predetermined time interval. For example, the method <NUM> may comprise providing the modified portions of the image data 308a to 308c, and the unselected portions of the image data 312a to 312e to a user at a first time. The modified 308a to 308c and unselected portions of image data 312a to 312e may be provided in the form of the output video stream <NUM> as described above in relation to <FIG>. The second data <NUM> may not be provided to the user at the first time but may be stored in memory.

Providing the portions of the image data 308a to 308c and 312a to 312e to the user may include transmitting the image data 308a to 308c and 312a to 312e to a data processing system being operated by the user, which is different to the data processing system <NUM> implementing the method <NUM>. Alternatively, the portions of the image data 308a to 308c and 312a to 312e may be provided to the user locally at the data processing system <NUM> implementing the method <NUM>. For example, the data processing system <NUM> may implement a computer program allowing users to review the output video stream <NUM>. The user may be assigned access credentials which allow them access to certain video streams such as the output video stream <NUM>.

At a second time that is after the first time, a request to provide the second data <NUM> to the user is received. This request may be received in response to the user trying to access the unredacted video stream <NUM>. On receipt of the request, the method <NUM> includes comparing the expiry date value, included in the metadata of the second data <NUM>, with a reference date value representing a current date and/or time. A predetermined action is performed in response to a determination of whether the expiry date value represents an earlier date and/or time than the reference date value or whether the expiry date value represents a later date and/or time than the reference date value.

If the expiry date value represents a later date and/or time than the reference date value, then the method <NUM> includes providing the second data <NUM> to the user. If the expiry date value represents an earlier date and/or time than the reference date value, then the method <NUM> includes responding to the request with a message indicating access to the second data <NUM> has expired. The reference date value may be stored in the data processing system <NUM> or may be obtained from a further data processing system which stores an objective reference date value. In this way, it is possible to allow access to the second data <NUM> for the user for a limited time. Control over the period of time for which second data <NUM> is made available to a user is useful in a variety of cases, one such example of this is where the video stream <NUM> is provided to a user under the condition of a warrant issued by a court. In this case, the court may specify that the video stream is made available to the user, who in this case may be an investigator, for a limited period of time to perform an investigation.

The method <NUM> may further comprise generating third data for use in recovering at least some of the selected portions of the image data from the modified portions of the image data 308a to 308c. <FIG> shows an example in which the data processing system <NUM> generates the third data <NUM>. The third data <NUM> may be suitable for recovering at least one different selected portion of the image data than can be recovered using the second data <NUM>. To this end the third data <NUM>, like the second data <NUM>, may include differences between certain modified portions of the image data 308a to 308c and selected portions of the image data. For example, the third data <NUM> may comprise differences to be used to recover a third set of the selected portions of the image data, and the second data <NUM> may comprise differences to be used to recover the first and/or second sets of the selected portions of the image data, which may be different from the third set of the selected portions. The third data <NUM> may also be used to recover at least some of the same selected portions as the second data <NUM>, in other words, the third set of selected portions may at least partially overlap with the first and/or second sets of selected portions of the image data.

Generating different second data <NUM> and third data <NUM> for use in recovering different selected portions of the image data allows the relevant data for recovering selected portions to be transmitted to the appropriate users, depending on their respective authorization. For example, a first user may be authorised to recover selected portions based on the third data <NUM>, a second user may be authorised to recover selected portions based on the second data <NUM>, and other users may be authorised to recover selected portions based on both or neither of the second data <NUM> and third data <NUM>. By providing either of the second data <NUM> or third data <NUM> to a user depending on their relative authorization, it is possible to reduce the amount of data which is transmitted to users and hence reduce bandwidth requirements. The third data <NUM> and the second data <NUM> may be encrypted according to different respective security traits, for example the first security trait and the second security trait described above. Alternatively, or additionally, some parts of the third data <NUM> may be encrypted according to different security traits than other parts of the third data <NUM>, similar to the examples described above in relation to the second data <NUM>.

The method <NUM> may further comprise keeping records of when the selected portions of the image data are recovered using the second data <NUM> and/or which users recovered the selected portions of the image data using the second data <NUM>. As discussed above, the selected portions of the image data may relate to sensitive information which is captured in the video stream <NUM>. Keeping a record of when and by whom the selected portions of the image data are recovered increases the control and security provided by the method <NUM>. To this end, the method <NUM> may include receiving an indication of a time at which the second data <NUM> is decrypted and an indication of a user who has accessed the decrypted second data <NUM>. A record may be stored comprising at least an indication of time at which the second data <NUM> was decrypted and the user who accessed the decrypted second data <NUM>. This indication may be received from a software module or computer program running on the data processing system <NUM> through which the user is accessing the output video stream <NUM> or may be received from a separate data processing system which is being operated by the user.

As discussed above in relation to <FIG>, providing second data <NUM> and third data <NUM> which are able to recover different selected portions of the image data allows different versions of the output video stream <NUM> to be viewed by different users depending on their respective authorization levels. To this end, the method <NUM> may further comprise providing the modified portions of the image data 308a to 308c, the unselected portions of the image data 312a to 312e, and the second data <NUM> to a first user and providing the modified portions of the image data 308a to 308c, the unselected portions of the image data 312a to 312e, and the third data <NUM> to a second user who is different to the first user.

Providing the portions of the image data 308a to 308c and 312a to 312e, and the second data <NUM> or third data <NUM> to users in this context may include transmitting this data to respective data processing systems operated by each of the first and second users. Alternatively, providing the portions of image data 308a to 308c and 312a to 312e and the second data <NUM> or third data <NUM> may include providing this data via a software module or computer program implemented on the data processing system <NUM> in dependence on an authority level assigned to each of the first and second users. For example, the first user and the second user may access a computer program on the data processing system <NUM> using credentials uniquely assigned to them. The data processing system <NUM> may be configured to provide access to the second data <NUM> to the first user and the third data <NUM> to the second user depending on their respective credentials.

It is to be appreciated by one skilled in the art that, while second data <NUM> and third data <NUM> have been described above, in some examples there may be further output data for recovering respective selected portions of the image data. For example, the method <NUM> may comprise generating fourth data, fifth data, and so forth, each of which can be used to recover different selected portions of the image data. In some cases, each recipient of the output video stream <NUM> may be provided with respective output data which is for use in recovering respective selected portions of the image data from modified portions of the image data 308a to 308c. This allows more granular control over the selected portions which each recipient is able to recover. Any number of the above-mentioned security traits can be used to encrypt the output data which is for use in recovering selected portions of the image data.

It is also to be appreciated by one skilled in the art that, while certain examples described herein relate to methods in which the second data <NUM> comprises differences between selected portions of the image data and modified portions of the image data 308a to 308c, the second data <NUM> may instead comprise replacement image data. In one such example the second data <NUM> comprises replacement pixels for the pixels in the modified portions of the image data 308a to 308c. In this case, the replacement pixels may be generated from the selected portions of the image data, e.g. the replacement pixels may be pixels included in the selected portions of the image data. When recovering the selected portions of the image data, pixels in the modified portions of the image data 308a to 308c may be replaced with the replacement pixels comprised in the second data <NUM>.

In the examples described above with respect to <FIG> and <FIG>, the selected portions of the image data which are processed to create the modified portions of the image data 308a to 308c are selected for every frame of the image data in which the sensitive information to be removed is present. However, in other examples, it may be sufficient to modify a subset of the plurality of frames of the image data that include the sensitive information which is to be removed. That is to say, the selected portions of the image data may correspond to a subset of the plurality of frames of image data 304a to <NUM> in the video stream <NUM> even when more frames of the image data in the video stream <NUM> contain the sensitive information. The examples described in relation to <FIG> and <FIG> rely on modifying the selected portions of the image data before encoding the unselected portions of the image data 312a to 312e and modified portions of the image data 308a to 308c. <FIG> shows an example of the method <NUM> in which portions of the image data are selected and the video stream is encoded before modifying the selected portions of the image data.

<FIG> shows a first module <NUM> for selecting portions of the image data in the input video stream <NUM> on the basis of the first data <NUM>. The selected portions of the image data are identified in selection data <NUM> representing at least some of the selected portions of the image data. The video stream <NUM> is encoded at a module <NUM> to generate an encoded video stream <NUM>. In some video encoding standards, such as the H. <NUM> video encoding standard, an encoded video stream may include a set of index frames of image data Ii and I<NUM>, representing a subset of the plurality of frames of image data in the input video stream <NUM>, and a set of intermediate frames of image data Di to D<NUM>. The intermediate frames of image data Di to Ds each include a set of difference vectors representing the differences between an index frame of image data Ii and the respective intermediate frame of image data Di to D<NUM> before the next sequential index frame I<NUM>. In the example shown in <FIG> there are two index frames I<NUM> and I<NUM> and between each index frame there are intermediate frames of image data Di to Ds represented by the set of difference vectors. This allows a reduced amount of image data to be included in the encoded video stream, while allowing the original video stream to be recovered. This is because parts of the images which don't change between the frames of image data do not need to be included in the intermediate frames of image data Di to Ds thereby reducing the amount of data included in the encoded video stream to represent certain frames of image data.

The index frames I<NUM> and I<NUM> are used periodically and, in some cases, may be used for specific frames of image data where the amount of difference between the specific frame of image data and a previous, most recent, index frame exceeds a predetermined threshold. At a module <NUM>, the data processing system <NUM> uses the selection data <NUM> to modify the selected portions of the image data in the encoded video stream <NUM>. In this case, modifying the selected portions of the image data includes modifying selected portions of the image data in the index frames I<NUM> and I<NUM> and not in the intermediate frames of image data Di to D<NUM>. Modifications made to an index frame of image data I<NUM> and I<NUM> by introducing noise or replacing pixel values will cause subsequent frames of image data, represented at Di to Ds, between the index frames of image data to also include these modifications when the encoded video stream is decoded. This is because the data values included in the frames of image data which are not the index frames specify differences between the given frame and an index frame rather than defining absolute data values which can be processed individually. By modifying the selected portions of the image data in the index frames I<NUM> and I<NUM> it allows fewer modifications to be made whilst still causing the sensitive information to be redacted in the output video stream <NUM>. In some cases, the sensitive information in the plurality of frames of image data which is to be removed by modifying selected portions of the image data relates to objects or people which are stationery in the scene captured in the video. In this case the method described with respect to <FIG> provides an efficient way of redacting the video stream without requiring that every frame of image data is processed. However, in other cases sensitive information, such as images of people in the video stream <NUM> may move between successive frames of image data. As such it may be desirable to increase the number of index frames I<NUM> and I<NUM> in the encoded video stream <NUM> such that the sensitive information which is to be redacted can be tracked between successive frames. The second data <NUM> is generated at the module <NUM> in a similar manner to the examples described in relation to <FIG> and <FIG>. That is to say the second data <NUM> may be derived from difference data, or may include the selected portions of the image data which are modified. <FIG> shows a non-transitory computer-readable storage medium <NUM> comprising computer-executable instructions <NUM> to <NUM> which, when executed by at least one processor <NUM>, causes the at least one processor <NUM> to perform the computer-implemented method <NUM> according to any of the examples described above.

<FIG> shows a data processing system <NUM> for image processing, which allows a user to recover the selected portions of the image data. The data processing system <NUM> comprises at least one processor <NUM> and storage <NUM>. The at least one processor <NUM> and the storage <NUM> include any suitable processor and storage combinations such as those described above in relation to the data processing system <NUM> shown in <FIG>. The data processing system <NUM> may be implemented as a single device or across multiple devices as a distributed data processing system <NUM>. For example, the data processing system <NUM> may be included as part of a cloud computing network which is accessible to a plurality of users through a local or wide area network.

The storage <NUM> comprises computer-readable instructions <NUM> which, when executed by the processor <NUM> cause the processor <NUM> to perform a method <NUM> for processing image data, which will now be described with reference to <FIG>. <FIG> shows a flow diagram illustrating the computer-implemented method <NUM>, and <FIG> show block diagrams of example implementations of the computer-implemented method <NUM>. The method <NUM> involves, at a first block <NUM>, receiving second data <NUM>, including for example, differences between the modified portions of the image data 308a to 308c and the selected portions of the image data, and modified portions of image data 308a to 308c generated according to the examples described in relation to any of <FIG>. At a second block <NUM>, the method involves processing at least some of the modified portions 308a to 308c using the second data <NUM>, e.g. the differences, to recover selected portions of the image data 1102a to 1102c.

The second data <NUM> and the modified portions of the image data 308a to 308c may be received in the form of a video stream, such as the output video stream <NUM>. In this case, the data processing system <NUM> comprises a decoder <NUM> for decoding the video stream. In examples where at least part of the second data <NUM> is encrypted, processing the modified portions of the image data 308a to 308c involves using encryption key data <NUM> to decrypt the at least part of the second data <NUM> which is encrypted. Where the data processing system <NUM> receives the output video stream <NUM>, the data processing system <NUM> may be further configured to reconstruct the original video stream <NUM>. For example the data processing system <NUM> may recover the selected portions 1102a to 1102c, using the modified portions of the image data 308a to 308c and the second data <NUM>, and reconstruct the video stream <NUM> using the unselected portions 312a to 312e and the selected portions 1102a to 1102c which have been recovered.

As described above, different parts of the second data <NUM> may be encrypted according to different security traits. The encryption key data <NUM> may represent a plurality of encryption keys each of which may be associated with a security trait, such that each of the encryption keys enables a different part of the second data <NUM> to be decrypted. Different users may be provided access to different sets of encryption keys based on an authorization associated with that user. For example, each user may be provided with an account (or log-in) which enables them to access none, some, or all of the encryption key data <NUM> to decrypt respective parts of the second data <NUM>.

The data processing system <NUM> may be included in a device which comprises other components suitable for processing a video stream and displaying a video. For example, a computing device comprising the data processing system <NUM> may be configured to reconstruct the original video stream <NUM> from the unselected portions of image data 312a to 312e of the output video stream <NUM>, selected portions of the image data, recovered using the data processing system <NUM>, and in some cases at least one of the modified portions of the image data 308a to 308c. The computing device may comprise any suitable display devices, video codecs, and so forth in order to display the reconstructed video to a user. In examples where a user does not have access to encryption key data <NUM>, e.g. due to not having the relevant authorization, the data processing system <NUM>, or a computing device comprising the data processing system <NUM>, may still be able to show the output video stream <NUM> to the user, for example by decoding and processing the output video stream <NUM>. However, the modified portions of the image data 308a to 308c will be displayed instead of the selected portions of the image data. In other words, a user without access to the encryption key data <NUM> will be able to view only a modified version, e.g. a redacted version, of the original video stream <NUM>.

Where the second data <NUM> includes metadata comprising an expiry date value, the method <NUM>, implemented by the data processing system <NUM>, may include obtaining a reference date value and comparing the reference date value to the expiry date value to determine whether the second data <NUM> can validly be used to recover the selected portions of the image data from the modified portions of the image data 308a to 308c. That is to say, processing at least some of the modified portions of the image data 308a to 308c using the second data <NUM> to recover the selected portions of the image data is conditional on the reference date value representing an earlier date and/or time than the expiry date value. In this way, the second data <NUM> is configured to expire, such that attempts by users to access the unredacted video stream <NUM> by recovering the selected portions of the image data can be restricted to a limited period of time.

Obtaining the reference date value may including reading the reference date value from storage at the data processing system <NUM>, wherein the data processing system maintains a reference date value representing a current date and/or time. Alternatively, the reference date value may be requested from a remote computing device which is separate to the data processing system <NUM>. For example, the reference date value may be obtained from an independent third-party computing system which stores an objective indication of the current date and/or time. Obtaining the reference date value from a remote computing device allows an objective determination to be made as to whether access to the second data <NUM> has expired. This in turn increases security by preventing users from circumventing the expiry date of the second data <NUM> by modifying the date and/or time stored at the data processing system <NUM>.

In the example shown in <FIG>, the data processing system <NUM> is configured to process the output video stream <NUM> and second data <NUM> as generated according to the examples described above in relation to <FIG>. In other words, in the examples shown in <FIG>, the data processing system <NUM> uses standards compliant decoding techniques, e.g. decoding techniques implemented by a general H. <NUM> decoder. The encoded and encrypted second data <NUM>, e.g. comprising differences between the selected portions of the image data and the modified portions of the image data 308a to 308c, is received and provided to a decrypting block <NUM> where is it is decrypted e.g. using encryption key data <NUM>. The decrypted second data <NUM> is then forwarded to a decoder, e.g. a standards compliant decoder, where it is decoded. The output video stream <NUM> is received and decoded at a decoder <NUM>, e.g. a standards compliant decoder. The decoded output video stream <NUM> and the decrypted and decoded second data <NUM> are then combined at block <NUM>. For modified portions of the image data 308a to 308c in the video stream <NUM>, the differences included in the second data <NUM> are used to recover the selected portions of the image data. By combining the second data <NUM> and the output video stream <NUM> in this way the original video stream <NUM> may be reconstructed.

As described above, in some examples, different parts of the second data <NUM> may be encrypted according to different security traits. In this case, the data processing system <NUM> may be configured to decrypt <NUM> only part of the second data <NUM>. Accordingly, parts of the second data <NUM>, e.g. differences between selected portions and modified portions of the image data 308a to 308c, which remain encrypted after the decryption block <NUM>, may not be combined <NUM> with the output video stream <NUM>. As such, in some examples only a subset of the selected portions of the image data may be recovered. Where the data processing system <NUM> does not have access to any encryption key data <NUM>, the differences in the second data <NUM> may not be combined with the output video stream <NUM>. Hence, the data processing system may only decode <NUM> the output video stream <NUM> to be displayed using a suitable media player.

The block diagram shown in <FIG> relates to examples of the method <NUM> in which modifications are made to standards compliant decoding methods, in particular, where the selected portions of the image data are macroblocks, as described above in relation to <FIG>.

In the example shown in <FIG>, an encrypted data packet <NUM>, comprising second data <NUM>, and an encoded modified macroblock <NUM> are received. At a first block <NUM> the data processing system <NUM> determines whether there is an encrypted macroblock in the encrypted data packet <NUM>. If there is an encrypted macroblock then then the encrypted macroblock is provided to a decryption block <NUM> where it is decrypted, dependent on there being suitable encryption key data <NUM> available.

The encoded modified macroblock <NUM> is decoded using a process which is similar to standards-compliant techniques as specified in the H. <NUM> AVC standard. In this example, the use of techniques such as entropy decoding <NUM>, inverse transforming <NUM>, and generating inter/intra frame decisions <NUM>, based on intra frame predictions <NUM> and motion compensation <NUM>, are used to decode the encoded modified macroblock <NUM>. Similarly, the decrypted output of block <NUM> is inversely transformed <NUM> and the selected macroblock is recovered based on the decoding techniques described above and the decrypted output of block <NUM>, e.g. differences between the selected macroblock and a respective modified macroblock. At a final block <NUM> a selection is made between the selected macroblock, generated based on the encrypted data packet <NUM>, and a decoded modified macroblock, based on the encoded modified macroblock <NUM>. Generally, where a recovered selected macroblock is available it is selected at <NUM>. If no recovered selected macroblock is available, for example, where the encryption key data <NUM> is not suitable or the encrypted data packet <NUM> does not include differences between some of the selected macroblocks and the modified macroblocks, then the decoded modified macroblock is selected and output at <NUM>. In this way, a user may be provided with an unredacted, or at least partially redacted, version of the original video stream <NUM> depending on encryption key data <NUM> which is available to the data processing system <NUM>.

Where the video stream <NUM> is processed according to the examples shown in <FIG>, processing at least some of the modified portions using the second data to recover selected portions of the image data is performed before decoding the output video stream <NUM>. In this way, the modifications made to the index frames I<NUM> and I<NUM> are reversed before decoding the video stream <NUM> such that the information recovered in the index frames I<NUM> and I<NUM> enable the sensitive information to also be recovered in the intermediate frames Di to Ds.

<FIG> shows a non-transitory computer-readable medium <NUM> comprising computer-executable instructions <NUM> and <NUM> which when executed by at least one processor <NUM>, cause the at least one processor <NUM> to perform a method <NUM> described above in relation to <FIG>.

While the examples described above relate to the modification of the selected portions of the image data by blurring, replacing, and/or redacting the selected portions of the image data, other examples are also envisaged. One such alternative comprises the use of encryption methods to modify the selected portions of the image data. <FIG> shows a block diagram of a method, which may be performed by the data processing system <NUM> shown in <FIG>, in which modifying the selected portions of the image data comprises encrypting the selected portions of the image data. An encoded macroblock <NUM> of image data is received and entropy decoded at block <NUM>. At block <NUM>, first data <NUM> is used to determine whether the macroblock <NUM> of image data should be selected. If the macroblock <NUM> is selected, then it is forwarded to block <NUM> where it is modified by being encrypted, to generate a modified macroblock <NUM>. The selected macroblock <NUM> may be encrypted according to a specific security trait, for example using a specific set of one or more encryption keys. In some examples, each selected macroblock is encrypted using a unique set of one or more encryption keys. Second data <NUM>, indicating a location of the selected macroblock <NUM> with respect to unselected macroblocks, is generated at block <NUM>. As encoding macroblocks relies on predictions generated from neighbouring macroblocks, adjustments are made at macroblock adjustment block <NUM>, to correct for errors which may otherwise be introduced by attempting to generate macroblock predictions from encrypted macroblocks, before the modified macroblock <NUM> and neighbouring macroblocks are entropy encoded at block <NUM>.

<FIG> shows a block diagram of a method of recovering the selected portions of the image data, in this case macroblocks, from the output of the method illustrated in <FIG>. The method illustrated in <FIG> may be performed by the data processing <NUM> shown in <FIG>. Encoded macroblocks are received and location information, included in the second data <NUM>, is used at block <NUM> to determine which encoded macroblocks are to be decrypted.

If a received encoded macroblock is to be decrypted, then it is forwarded to block <NUM> where it is decrypted using encryption key data <NUM>. Encoded macroblocks which are not to be decrypted, e.g. unencrypted macroblocks, are forwarded to a macroblock adjustment block <NUM> which also receives the decrypted macroblocks. Adjustments are made between the decrypted macroblocks and those which are not to be decrypted and the output of this stage is forwarded to a decoder <NUM>, which decodes the macroblocks such that they can be viewed in a media player.

Where modifying macroblocks comprises encrypting the macroblocks, adjustments to the encoding process may be made to mitigate changes in noise and errors which may be introduced. Errors can occur when trying to use encrypted macroblocks, which cannot be read, when generating predictions for encoding neighbouring macroblocks. Adjusting the encoding process can prevent such errors from arising and may include, for example, using the first data <NUM> to inform motion estimation operations included in an encoding process to adjust for errors introduced by the use of encrypted macroblocks, e.g. when performing motion estimation the adjustment may include restricting the selection of modified macroblocks when performing motion estimation. The first data <NUM> may also be used to compensate for the encrypted macroblocks in an inter/intra frame prediction decision as discussed above in relation to <FIG>.

The processing of at least some of the modified portions of the image data 308a to 308c using the second data <NUM> may be performed in response to a request from a user. In this case, the method <NUM> implemented by the data processing system <NUM> may further comprise storing a record which includes an indication of a time at which the modified portions of the image data 308a to 308c are processed using the second data <NUM> and an indication of the user. In this way, an audit trail of users which have recovered the selected portions of the image data may be maintained which increases the security and accountability of the users.

Claim 1:
A computer-implemented method for processing a video stream (<NUM>), the computer-implemented method comprising:
receiving (<NUM>) a video stream (<NUM>) comprising a plurality of frames of image data (304a to <NUM>);
selecting (<NUM>) portions of the image data on the basis of first data (<NUM>) indicating locations within the image data;
modifying (<NUM>) the selected portions to generate modified portions of the image data (308a to 308c);
generating (<NUM>) second data (<NUM>) for use in recovering at least some of the selected portions from the modified portions (308a to 308c), the second data (<NUM>) comprising metadata representing an expiry date value for the second data (<NUM>);
outputting (<NUM>) the second data (<NUM>), unselected portions of the image data and the modified portions of the image data (312a to 312e);
providing the modified portions of the image data (308a to 308c) and the unselected portions of the image data (312a to 312e) to a user at a first time; and
at a second time, the second time being after the first time:
receiving a request to provide the second data (<NUM>) to the user;
comparing the expiry date value for the second data (<NUM>) with a reference date value; and
performing a predetermined action in dependence on the comparison of the expiry date value with the reference date value.