Patent Description:
In representation learning, a neural network can learn to transform a complex input, such as an image, text or sound, into a representation vector. The representation vector is to be used by a classifier, such as a standard linear classifier, to solve various classification problems.

The representation vector can encode various human-understandable concepts that are embedded in the complex input. A "concept" can be a binary attribute of a class of instances. For example, binary attribute [<NUM>, <NUM>] can respectively encode "Male" and "not Male" for the concept class "Male", [<NUM>] representing that "not Male" is an instance outside of the concept class "Male". In the same example, the binary attribute "Male" and "not Male" can also be associated with the concept class "Gender". In another example, binary attribute [<NUM>, <NUM>] can respectively encode "Young" and "not Young" is associated with the concept the concept class "Young", [<NUM>] representing that "not Young" is an instance outside of the concept class "Young". In the same example, the binary attribute "Young" and "not Young" can also be associated with the concept class "Age".

In some instances, the representation vector can encode useful concepts, where the concepts are relevant to the classification task. For example, the representation vector could encode concept class "Stripes" when the neural network classifies whether an image comprises a "Zebra". For example, the representation vector could encode concept class "Clothing" when a neural network classifies whether an image comprises an "Apron". In another example, the representation vector could encode concept class "Stethoscope" when a neural network classifies whether a person is a "Nurse" or a "Doctor". In another example, the representation vector could encode concept class "Sphere" when a neural network classifies whether an image comprises a "Ping-Pong Ball".

However, in some instances, the representation vector could encode undesirable, or even malicious, concepts, where the concepts relate to user sensitive information and/or misleading information. For example, some concepts may represent detrimental features, such as ones that are not relevant to the downstream task, but are nevertheless spuriously correlated with the target variable, e.g., the background for classifying the type of animal; some of the attributes might represent information that was once informative but nonetheless is no longer so; others may represent sensitive features, such as gender or race, which are undesirable for the model to correlate with. The user sensitive information and/or misleading information should not be considered when the neural network is completing a classification task. This is because the representation vector may encode concepts that can result in spurious correlations between the target variables and the undesirable concept.

For example, the representation vector could encode concept class "Trees" when the neural network classifies whether an image comprises a "Zebra". For example, the representation vector could encode concept class "Gender" when a neural network classifies whether an image comprises an "Apron". In another example, the representation vector could encode concept class "Gender" when a neural network classifies whether a person is a "Nurse" or a "Doctor". In another example, the representation vector could encode concept class "Race" when a neural network classifies whether an image comprises a "Ping-Pong Ball". All the examples reveal the model's reliance on sensitive information (gender and race) or misleading information (trees).

<NPL>" discloses that "Neural network models trained on text data have been found to encode undesirable linguistic or sensitive concepts in their representation. Removing such concepts is non-trivial because of a complex relationship between the concept, text input, and the learnt representation. Recent work has proposed post-hoc and adversarial methods to remove such unwanted concepts from a model's representation". There is therefore a desire to detect and remove, or at least reduce, the reliance on an undesirable concept by a neural network, whilst retaining other useful concepts. In more general terms, there is a desire to provide a neural network with improved out-of-distribution (OOD) generalization and distributionally robust optimization (DRO). Removing the undesirable concepts will produce more robust, generalizable and fair models that are oblivious to the presence of them.

The embodiments described below are not limited to implementations which solve any or all of the disadvantages of the known approaches described above.

This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to determine the scope of the claimed subject matter.

According to a first aspect of the invention, there is provided a system for removing a concept from a trained neural network for executing a classification task on an input corrupted data item, wherein the input corrupted data item is an image or video, and an output of the classification task is a classification of an object present in the image or video, the system comprising:.

the trained neural network, wherein the trained neural network comprises a hidden layer; and a plurality of classifiers respectively incorporated in a plurality of layers of the hidden layer, wherein: the hidden layer comprises a plurality of contracting layers and respectively a plurality of layers preceding the contracting layers, wherein the plurality of contracting layers have smaller dimensions than the plurality of layers preceding the contracting layers; the plurality of classifiers are respectively incorporated in the plurality of layers preceding the contracting layers; each classifier defines a representation vector at the layer of the hidden layer, wherein the concept is relevant to the classification task and the representation vector classifies instances of the concept and non-instances of the concept at the layer; each classifier defines a concept activation vector, wherein the concept activation vector is a normal vector to the representation vector and the concept activation vector comprises an adversarial penalty objective to reduce the instances of the concept at the layer; and a loss function of the trained neural network is optimised based on a downstream loss of the classification task and the adversarial penalty objective.

In an embodiment, the hidden layer comprises a contracting layer and a layer preceding the contracting layer, wherein the contracting layer has smaller dimensions than the layer preceding the contracting layer.

In an embodiment, the hidden layer comprises a plurality of contracting layers and a plurality of layers preceding the contracting layers, wherein the plurality of contracting layers have smaller dimensions than the plurality of layers preceding the contracting layers; and the plurality of classifiers are respectively applied to the plurality of layers preceding the contracting layers.

In an embodiment, the classifier has been trained on a concept dataset, wherein the concept dataset comprises examples of the concept instances and examples non-concept instances.

In an embodiment, the examples of the concept instances and examples non-concept instances are out-of-distribution of a training data of the classification task.

In an embodiment, the examples of the concept instances are abstract examples of the concept unrelated to the training data of the classification task and the examples non-concept instances are random data items.

In an embodiment, the classifier is a linear adversarial classifier.

In an embodiment, the adversarial penalty objective is adversarial penalty = γ∥lvC,k,λ(W)∥<NUM> , wherein: γ is a scaling factor; and vC,k,λ(W) is the concept activation vector, wherein: <MAT> is the binary cross-entropy loss, wherein ℓBCE(p,y) = -ylogp - (<NUM> - y)log (<NUM> - p); σ(x) is the sigmoid function, wherein σ(x) = <NUM>/<NUM>(<NUM> + e-x); θ is a parameter vector of the trained neural network; hk is the representation vector at the layer; W is the parameter of the representation; and fk(. ;θ) is the classifier applied at the layer of the trained neural network.

In an embodiment, the adversarial penalty objective is optimised with implicit gradients.

In an embodiment, the loss function is the sum of the downstream loss of the classification task and the adversarial penalty objective.

In an embodiment, the loss function is optimised by stochastic gradient descent.

According to a second aspect of the invention, there is provided a computer implemented method for removing a concept from a trained neural network for executing a classification task on an input corrupted data item, wherein the input corrupted data item is an image or video, and an output of the classification task is a classification of an object present in the image or video, wherein the trained neural network comprises a hidden layer, the method comprising: respectively incorporating a plurality of classifiers to a plurality of layers of the hidden layer, wherein: the hidden layer comprises a plurality of contracting layers and respectively a plurality of layers preceding the contracting layers, wherein the plurality of contracting layers have smaller dimensions than the plurality of layers preceding the contracting layers; and the plurality of classifiers are respectively incorporated in the plurality of layers preceding the contracting layers; defining a representation vector at the plurality of layers of the hidden layer, wherein the concept is relevant to the classification task and representation vector classifies instances of the concept and non-instances of the concept at the layer; defining a concept activation vector, wherein the concept activation vector is a normal vector to the representation vector and the concept activation vector comprises an adversarial penalty objective to reduce the instances of the concept; and optimising a loss function of the trained neural network based on a downstream loss of the classification task and the adversarial penalty objective.

According to a third aspect of the invention, there is provided a computer implemented method for training a neural network to remove a concept from the neural network, wherein the neural network is for executing a classification task on an input corrupted data item, wherein the input corrupted data item is an image or video, and an output of the classification task is a classification of an object present in the image or video, the method comprising: training a plurality of classifiers to define a representation vector at a respective plurality of layers of a hidden layer of the neural network, wherein the concept is relevant to the classification task and the representation vector classifies instances of the concept and non-instances of the concept at the layer, wherein training the classifier comprises: providing a concept dataset to each classifier, wherein the concept dataset comprises examples of concept class instances and examples of non-concept instances; and defining a concept activation vector, wherein the concept activation vector is a normal vector to the representation vector and the concept activation vector comprises an adversarial penalty objective to reduce the instances of the concept at the layer; respectively incorporating a plurality of classifiers to a plurality of layers of the hidden layer, wherein: the hidden layer comprises a plurality of contracting layers and respectively a plurality of layers preceding the contracting layers, wherein the plurality of contracting layers have smaller dimensions than the plurality of layers preceding the contracting layers; and the plurality of classifiers are respectively incorporated in the plurality of layers preceding the contracting layers.

According to a fourth aspect of the invention, there is provided a computer readable storage medium for removing a concept from a trained neural network when executed by one or more processing devices, the computer readable instructions causes the one or more processing devices to carry out the second or third aspect of the invention.

The methods described herein may be performed by software in machine readable form on a tangible storage medium e.g. in the form of a computer program comprising computer program code means adapted to perform all the steps of any of the methods described herein when the program is run on a computer and where the computer program may be embodied on a computer readable medium. Examples of tangible (or non-transitory) storage media include disks, thumb drives, memory cards etc. and do not include propagated signals. The software can be suitable for execution on a parallel processor or a serial processor such that the method steps may be carried out in any suitable order, or simultaneously.

This application acknowledges that firmware and software can be valuable, separately tradable commodities.

The preferred features may be combined as appropriate, as would be apparent to a skilled person, and may be combined with any of the aspects of the invention.

Common reference numerals are used throughout the figures to indicate similar features.

Embodiments of the present invention are described below by way of example only. These examples represent the best mode of putting the invention into practice that are currently known to the Applicant although they are not the only ways in which this could be achieved. The description sets forth the functions of the example and the sequence of steps for constructing and operating the example. However, the same or equivalent functions and sequences may be accomplished by different examples.

<FIG> shows a corrupted image <NUM> from an MNIST dataset, comprising MNIST digits with spuriously correlated stripes. The illustrated embodiments show a corrupted image <NUM>; however, the present invention is applicable to any form of data item. For example, video, audio, text, etc. This list is not intended to be exhaustive. Other data items are supported in some Notably, only the embodiments with videos and images are claimed.

The corrupted image <NUM> is a corrupted MNIST image, wherein the MNIST digits are undesirably correlated with stripes. Each digit of the corrupted MNIST image includes stripes within its background, wherein the angle of the stripes spuriously correlates to the MNIST digit. For example, all MNIST digit "<NUM>" has stripes at an angle "Angle A", all MNIST digit "<NUM>" has stripes at an angle "Angle B", all MNIST digit "<NUM>" has stripes at an angle "Angle C", etc. Accordingly, there is a spurious correlation between the angle of the stripes in the corrupted MNIST image and the digit in the corrupted MNIST image. This is because the angle of the stripes should have no relevance when a neural network is determining the numerical value of MNIST digit. Instead, the neural network should only take into account, for example, the size and shape of the MNIST digits. However, undesirably, a trained neural network may rely on the angle of the stripes when determining the MNIST digit from the corrupted image <NUM>. Therefore, it is desirable to remove the concept "Stripes" from a neural network trained to identify MNIST digits from the corrupted image <NUM>.

<FIG> shows a concept dataset <NUM> for removing the "Stripes" concept from a neural network trained to identify MNIST digits from the corrupted image <NUM>. <FIG> shows a neural network <NUM> implementing a linear adversarial classifier (LAC) <NUM>.

To remove the concept "Stripes" from a trained neural network trained to identify MNIST digits from the corrupted image <NUM>, a linear adversarial classifier (LAC) <NUM> is trained using a concept dataset <NUM> to determine the extent to which the neural network is influenced by the "Stripes" concept.

After training the LAC <NUM> using the concept dataset <NUM>, the LAC <NUM> defines a concept activation vector (CAV) <NUM> that determines whether the trained neural network is influenced by the "Stripes" concept. The CAV <NUM> is derived from a vector representation from a layer of the trained neural network, for example, a layer in a hidden layer of the trained neural network and/or a penultimate layer of the trained neural network.

The concept dataset <NUM> comprises examples of concept class instances <NUM> and examples of non-concept instances <NUM>. The concept dataset <NUM> of the present embodiment comprises examples of "Stripes" concept class instances and examples of non-"Stripes" concept class instance. <FIG> shows the top five rows including examples of the concept class "Stripes" and the bottom five rows show examples that do not belong to the concept class "Stripes".

In the illustrated embodiment of <FIG>, the concept dataset <NUM> comprises examples of the concept class "Stripes" and examples that do not belong to the concept class "Stripes", both from the EMNIST (not MNIST) letters. The examples of concept class instances <NUM> can be abstract examples of the concept. The examples of non-concept instances <NUM> can be random data items. In other words, the concept dataset <NUM> may not be directly related to the classification task that the neural network is trained to do. Instead, the concept dataset <NUM> can be out-of-distribution of the training distribution. This is advantageous because then the CAV will not be restricted by spurious correlations present in the training data. A concept dataset <NUM> comprising abstract concept class instances <NUM> and random non-concept instances <NUM> advantageously allows the use of pre-labelled public datasets including the undesirable concept. The concept dataset <NUM> can resemble the input of the trained neural network and still be out-of-distribution. For example, the illustrated embodiment shows that EMNIST digits resembles the input MNIST digits but are nevertheless out-of-distribution.

The concept dataset <NUM> can be expressed as <MAT>, wherein: C refers to the concept class; NC is the total number of examples in the concept dataset <NUM>; i is an index of an example of the concept dataset <NUM>; <MAT> are the instances of the concept dataset <NUM>; and <MAT> and indicates whether the instance <MAT> is from the concept class, or not.

The neural network has a prediction model in the form P (Y | X), wherein Y is an output of the trained neural network and X is the input of the trained neural network. The trained neural network has the following equation P (Y | X) = fk(hk(X;W);θ).

θ is a parameter vector of the trained neural network. hk is a representation vector of the kth layer of the trained neural network, which is also herein referred to as the representation. W is the parameter of the representation. ;θ) is a classifier applied at the kth layer of the trained neural network.

<FIG> shows a concept activation vector <NUM> of the linear adversarial classifier <NUM>. The CAV <NUM> of the LAC <NUM> is denoted as vc,K,λ(W). The CAV <NUM> is a normal vector to the hyperplane separating the corresponding representations (concept separation hyperplane), i.e. the instances of the concept class and non-instances of the concept class. The sensitivity of the trained neural network to the concept class is calculated by taking a directional derivative of the CAV <NUM>. The sensitivity of the trained neural network to the concept class can be used to check the extent to which the concept class is relied on for the classification task. For example, in this instance, the stripe class for the classification of the MNIST digits in the corrupted image <NUM>. Therefore, a change in the in the sensitivity of the trained neural network to the concept class can be observed in response to the input X in the direction of the CAV <NUM>. The sensitivity of the trained neural network to the concept class has the following equation <MAT>.

The CAV <NUM> comprises a penalized logit regression. The CAV <NUM> has the following equation <MAT>.

ℓBCE is the binary cross-entropy loss, wherein ℓBCE(p,y) = -ylogp - (<NUM> - y)log(<NUM> - p). σ(x)is the sigmoid function, wherein σ(x) = <NUM>/<NUM>(<NUM> + e-x).

The CAV <NUM> of the LAC <NUM> is penalized to encourage the neural network to be less influenced by the undesirable concept class. The LAC <NUM> is trained using an implicit differentiation technique. This improves stability of adversarial training. Therefore, the trained neural network can optimise the following objective with implicit gradients.

The CAV has the following adversarial penalty objective adversarial penalty = γ∥vC,k,λ(W)∥<NUM> , wherein γ is a scaling factor. Therefore, the loss function of the neural network is Total Loss = Downstream Loss + Adversarial Penalty.

The downstream loss is a loss from the downstream task and depends on the input data of the trained neural network, such as, in this example, the classification of stripped MNIST digits. The total loss can be optimised by various optimisation methods, such as stochastic gradient descent. The skilled person would understand that other gradient descent algorithms can be used for the optimisation method, such as stochastic gradient with momentum and ADAM.

A linear adversarial classifier (LAC) is particularly advantageous over other adversarial classifiers because concepts are often associated with linear directions in a latent space. A LAC is also particularly advantageous over other adversarial classifiers because, due to the differences in the data sources for the downstream task (MNIST in this example) and the concept dataset (EMNIST in this example), a LAC is more likely to generalize between the two datasets. LAC is also particularly well suited for optimising the classification task with implicit gradients. However, it is not essential that a LAC is used to remove an undesirable concept from the trained neural network. The skilled person would understand that other classifiers, such as non-linear classifiers, or standard adversarial training methods could be used to obtain the CAV <NUM>.

<FIG> shows a trained neural network <NUM> implementing a LAC <NUM> at a penultimate layer <NUM> of the trained neural network <NUM>. The trained neural network <NUM> comprises an input layer <NUM>, a hidden layer <NUM>, a penultimate layer <NUM>, an output layer <NUM>.

The trained neural network <NUM> is trained to classify the value present in the instance of the corrupted image <NUM>. The trained neural network <NUM> receives an instance of a corrupted image <NUM> at the input layer <NUM>. Subsequent hidden layers <NUM> receive a signal from a preceding hidden layer or input layer and output a corresponding signal depending on the weights of each node of each hidden layer <NUM>. The penultimate layer <NUM> of the trained neural network <NUM>, receives a signal from a preceding hidden layer and outputs a corresponding signal depending on the weights of each node of the penultimate layer <NUM>. The penultimate layer <NUM> outputs a signal to the output layer <NUM>, which determines the classification of the value present in the instance of the corrupted image <NUM>, i.e. <NUM> - <NUM> of the MNIST digits.

In the illustrated embodiment, a LAC <NUM> is applied to the penultimate layer <NUM> of the trained neural network <NUM>. The LAC <NUM> obtains a concept activation vector <NUM> of the output of the penultimate layer <NUM>. The CAV <NUM> is obtained by training the LAC <NUM> with the concept dataset <NUM>. The CAV <NUM> classifies whether the "Stripe" concept influenced the classification of the value present in the instance of the corrupted image <NUM>.

After the CAV <NUM> is obtained by the LAC <NUM> at the penultimate layer <NUM>, the adversarial penalty of the CAV <NUM> is calculated according to adversarial penalty = γ∥vC,k,λ(W)∥<NUM>. Accordingly, the total loss function of the current classification carried out by the trained neural network <NUM> is obtained according to Total Loss = Downstream Loss + Adversarial Penalty.

Thereafter, the trained neural network <NUM> is optimised to reduce the downstream loss and the adversarial penalty using an optimisation method, such as stochastic gradient descent. After the trained neural network <NUM> is optimised, the trained neural network removes the undesirable concept when carrying out the classification task.

<FIG> shows a trained neural network <NUM> implementing the linear adversarial classifier <NUM> at a hidden layer <NUM> of the trained neural network <NUM>. For simplicity, only the differences between the trained neural network <NUM> of <FIG> and the trained neural network <NUM> of <FIG> shall be explained.

Information about a concept propagates through the layers of the trained neural network <NUM>. As the information about the concept propagates through a single layer, the information is entangled with other layers and neurons of the trained neural network <NUM>. Accordingly, it is desirable to remove an undesirable concept before the concept entangles with other features in subsequent layers and neurons of the trained neural network <NUM>.

In order to remove an undesirable concept before the concept entangles with other features in subsequent layers, the LAC <NUM> is applied at a hidden layer <NUM> of the trained neural network <NUM>. The LAC <NUM> can be applied at a layer other than the input layer <NUM>, penultimate layer <NUM> or the output layer <NUM>. Advantageously, removing an undesirable concept before the undesirable concept entangles with other parts of the trained neural network improves robustness and out-of-distribution (OOD) generalization of the trained neural network <NUM> to complete the classification task.

Advantageously, the plurality LACs are applied to deeper layers in the neural network. This helps to improve transferability of the representations between examples of concept class instances and non-concept instances.

<FIG> shows a trained neural network comprising contracting layers and implementing the linear adversarial classifier at the penultimate layer of the trained neural network. For simplicity, only the differences between the trained neural network <NUM> of <FIG> and the trained neural network <NUM> of <FIG> shall be explained.

The trained neural network <NUM> of <FIG> comprises a contracting layer <NUM>. The trained neural network <NUM> further comprises a layer preceding contraction <NUM>. The contracting layer <NUM> and the layer preceding contraction <NUM> can be in a hidden layer of the trained neural network <NUM>. A layer preceding contraction <NUM> is a layer in the trained neural network <NUM>, wherein the contracting layer of the trained neural network <NUM> receiving the output of the layer preceding contraction <NUM> as input is smaller in dimension than the layer preceding contraction <NUM>. When the LAC <NUM> is applied at the penultimate layer <NUM> of a trained neural network <NUM> with contracting layers <NUM>, concept removal is further improved. The penultimate layer can also be a contracting layer <NUM> or a layer preceding contraction <NUM>, as shown in <FIG>.

The trained neural network <NUM> can comprise a layer preceding contraction <NUM> in the hidden layers <NUM> of the trained neural network <NUM>. The penultimate layer <NUM> can also be a layer preceding contraction <NUM>.

<FIG> shows a trained neural network <NUM> implementing a plurality of LACs at a plurality of layers in a hidden layer of the trained neural network <NUM>. For simplicity, only the differences between the trained neural network <NUM> of <FIG> and the trained neural network <NUM> of <FIG> shall be explained.

To further prevent an undesirable concept to propagate through the trained neural network <NUM>, a plurality of LACs <NUM> are incorporated in a plurality of layers in the hidden layer <NUM> of the trained neural network <NUM>. For example, the illustrated embodiment of trained neural network <NUM> shows that the plurality of LACs <NUM> are applied to hidden layers <NUM> and the penultimate layer <NUM>. The plurality of LACs <NUM> can work simultaneously to remove the concept from the trained neural network <NUM> at the plurality of layers in the hidden layer <NUM>.

Accordingly, where there is a plurality of LACs <NUM>, the neural network can optimise the following objective with implicit gradients. <MAT><MAT>.

km is the layer of a linear adversarial classifier. λm is the regularization parameter. M is the total number of linear adversarial classifiers. m is an index for the linear adversarial classifiers. γ is a scaling factor.

<FIG> shows a trained neural network comprising a plurality of layers preceding contraction <NUM> each respectively implementing a plurality of LACs <NUM>. For simplicity, only the differences between the trained neural network <NUM> of <FIG> and the trained neural network <NUM> of <FIG> shall be explained.

Advantageously, the LACs <NUM> are applied at the layer preceding contraction <NUM>. This significantly improves the effectiveness of removing an undesirable concept. It is not essential for the LACs to be applied to all layers preceding contraction <NUM>. The LACs can be applied to at least one layer preceding contraction <NUM>. The LACs <NUM> can be applied, in combination, at layers preceding contraction <NUM> and non-contracting layers, i.e. any other layer in the hidden layer <NUM> of the trained neural network <NUM>.

<FIG> shows a specific worked example of a trained neural network <NUM> according to an embodiment. A worked example of the embodiments set out above will now be described with reference to <FIG>.

<FIG> shows a ConvNet with the width of each layer corresponding to the total dimension of a flattened representation. The bottom number present at the first bottom layer corresponds to input dimension <NUM> = <NUM> x <NUM>. Each layer preceding contraction <NUM> of the trained neural network <NUM> are pointed with an arrow. The last layer, the output layer, of the trained neural network <NUM> is not shown for simplicity.

The trained neural network <NUM> was tested to classify the MNIST digit in corrupted image <NUM> showing the striped MNIST dataset. LACs <NUM> were applied to various combinations of layers of the trained network <NUM>. In the table below, for each combination we show train and test error percentages, with testing done on images without stripes, i.e. non-corrupted images. The table shows that applying the LAC <NUM> to layer preceding contraction <NUM> significantly improves the test results. The layer preceding contraction <NUM> in the table are marked with (p).

The embodiments described above are fully automatic. In some examples a user or operator of the system may manually instruct some steps of the method to be carried out.

In the described embodiments of the invention parts of the system may be implemented as a form of a computing and/or electronic device. Such a device may comprise one or more processors which may be microprocessors, controllers or any other suitable type of processors for processing computer executable instructions to control the operation of the device in order to gather and record routing information. In some examples, for example where a system on a chip architecture is used, the processors may include one or more fixed function blocks (also referred to as accelerators) which implement a part of the method in hardware (rather than software or firmware). Platform software comprising an operating system or any other suitable platform software may be provided at the computing-based device to enable application software to be executed on the device.

Various functions described herein can be implemented in hardware, software, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media may include, for example, computer-readable storage media. Computer-readable storage media may include volatile or non-volatile, removable or non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. A computer-readable storage media can be any available storage media that may be accessed by a computer. By way of example, and not limitation, such computer-readable storage media may comprise RAM, ROM, EEPROM, flash memory or other memory devices, CD-ROM or other optical disc storage, magnetic disc storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disc and disk, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and blu-ray disc (BD). Further, a propagated signal is not included within the scope of computer-readable storage media. Computer-readable media also includes communication media including any medium that facilitates transfer of a computer program from one place to another. A connection, for instance, can be a communication medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of communication medium.

For example, and without limitation, hardware logic components that can be used may include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc..

Although illustrated as a single system, it is to be understood that a computing device may be a distributed system. Thus, for instance, several devices may be in communication by way of a network connection and may collectively perform tasks described as being performed by the computing device. Although illustrated as a local device it will be appreciated that the computing device may be located remotely and accessed via a network or other communication link (for example using a communication interface).

The term 'computer' is used herein to refer to any device with processing capability such that it can execute instructions. Those skilled in the art will realise that such processing capabilities are incorporated into many different devices and therefore the term 'computer' includes PCs, servers, mobile telephones, personal digital assistants and many other devices.

Those skilled in the art will realise that storage devices utilised to store program instructions can be distributed across a network. For example, a remote computer may store an example of the process described as software. A local or terminal computer may access the remote computer and download a part or all of the software to run the program. Those skilled in the art will also realise that by utilising conventional techniques known to those skilled in the art that all, or a portion of the software instructions may be carried out by a dedicated circuit, such as a DSP, programmable logic array, or the like.

Variants should be considered to be included into the scope of the invention.

Any reference to 'an' item refers to one or more of those items. The term 'comprising' is used herein to mean including the method steps or elements identified, but that such steps or elements do not comprise an exclusive list and a method or apparatus may contain additional steps or elements.

As used herein, the terms "component" and "system" are intended to encompass computer-readable data storage that is configured with computer-executable instructions that cause certain functionality to be performed when executed by a processor. The computer-executable instructions may include a routine, a function, or the like. It is also to be understood that a component or system may be localized on a single device or distributed across several devices.

Further, as used herein, the term "exemplary" is intended to mean "serving as an illustration or example of something".

Further, to the extent that the term "includes" is used in either the detailed description or the claims, such term is intended to be inclusive in a manner similar to the term "comprising" as "comprising" is interpreted when employed as a transitional word in a claim.

The figures illustrate exemplary methods. While the methods are shown and described as being a series of acts that are performed in a particular sequence, it is to be understood and appreciated that the methods are not limited by the order of the sequence. For example, some acts can occur in a different order than what is described herein. In addition, an act can occur concurrently with another act. Further, in some instances, not all acts may be required to implement a method described herein.

Moreover, the acts described herein may comprise computer-executable instructions that can be implemented by one or more processors and/or stored on a computer-readable medium or media. The computer-executable instructions can include routines, sub-routines, programs, threads of execution, and/or the like. Still further, results of acts of the methods can be stored in a computer-readable medium, displayed on a display device, and/or the like.

The order of the steps of the methods described herein is exemplary, but the steps may be carried out in any suitable order, or simultaneously where appropriate. Additionally, steps may be added or substituted in, or individual steps may be deleted from any of the methods without departing from the scope of the subject matter described herein.

Claim 1:
A system for removing a concept from a trained neural network (<NUM>, <NUM>, <NUM>) for executing a classification task on an input corrupted data item (<NUM>), wherein the input corrupted data item is an image or video, and an output of the classification task is a classification of an object present in the image or video, the system comprising:
the trained neural network (<NUM>, <NUM>, <NUM>), wherein the trained neural network (<NUM>, <NUM>, <NUM>) comprises a hidden layer (<NUM>, <NUM>); and
a plurality of classifiers (<NUM>) respectively incorporated in a plurality of layers of the hidden layer (<NUM>, <NUM>), wherein:
the hidden layer (<NUM>, <NUM>) comprises a plurality of contracting layers (<NUM>) and respectively a plurality of layers preceding the contracting layers (<NUM>), wherein the plurality of contracting layers (<NUM>) have smaller dimensions than the plurality of layers preceding the contracting layers (<NUM>);
the plurality of classifiers (<NUM>) are respectively incorporated in the plurality of layers preceding the contracting layers (<NUM>);
each classifier (<NUM>) defines a representation vector at the layer of the hidden layer (<NUM>, <NUM>), wherein the concept is relevant to the classification task and the representation vector classifies instances of the concept and non-instances of the concept at the layer;
each classifier defines a concept activation vector (<NUM>), wherein the concept activation vector (<NUM>) is a normal vector to the representation vector and the concept activation vector (<NUM>) comprises an adversarial penalty objective to reduce the instances of the concept at the layer; and
a loss function of the trained neural network (<NUM>, <NUM>) is optimised based on a downstream loss of the classification task and the adversarial penalty objective.