INFO:rldata.rlpedata:Loading dataset from zh-plus/tiny-imagenet
Fast image processor class <class 'transformers.models.vit.image_processing_vit_fast.ViTImageProcessorFast'> is available for this model. Using slow image processor class. To use the fast image processor class set `use_fast=True`.
Some weights of ViTForImageClassification were not initialized from the model checkpoint at google/vit-base-patch16-224-in21k and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
INFO:model.modelloader:Successfully initialized transformers model: google/vit-base-patch16-224-in21k
INFO:art.estimators.classification.pytorch:Inferred 1 hidden layers on PyTorch classifier.
INFO:defences.defencerlpe:DefenseManager initialized with available defenses.
['SpatialSmoothing', 'FeatureSqueezing', 'JpegCompression']
INFO:attacks.attackrlpe:Initializing attack: CarliniL2Method
C&W L_2: 100%|███████████████████████████████████████████████████████████████████████████████████████| 32/32 [09:19<00:00, 17.49s/it]
INFO:art.attacks.evasion.carlini:Success rate of C&W L_2 attack: 0.00%
INFO:attacks.attackrlpe:Attack 'CarliniL2Method' performed successfully.
Traceback (most recent call last):
  File "/scratch/hemanth/Hemanth/RLPE/rlpe.py", line 112, in <module>
    model_output = wrapped_model(images)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/hemanth/anaconda/envs/rlpe/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/hemanth/anaconda/envs/rlpe/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/hemanth/Hemanth/RLPE/rlpe.py", line 59, in forward
    output = self.original_model(x)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/hemanth/anaconda/envs/rlpe/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/hemanth/anaconda/envs/rlpe/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/hemanth/anaconda/envs/rlpe/lib/python3.12/site-packages/transformers/models/vit/modeling_vit.py", line 840, in forward
    outputs = self.vit(
              ^^^^^^^^^
  File "/scratch/hemanth/anaconda/envs/rlpe/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/hemanth/anaconda/envs/rlpe/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/hemanth/anaconda/envs/rlpe/lib/python3.12/site-packages/transformers/models/vit/modeling_vit.py", line 619, in forward
    embedding_output = self.embeddings(
                       ^^^^^^^^^^^^^^^^
  File "/scratch/hemanth/anaconda/envs/rlpe/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/hemanth/anaconda/envs/rlpe/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/hemanth/anaconda/envs/rlpe/lib/python3.12/site-packages/transformers/models/vit/modeling_vit.py", line 124, in forward
    embeddings = self.patch_embeddings(pixel_values, interpolate_pos_encoding=interpolate_pos_encoding)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/hemanth/anaconda/envs/rlpe/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/hemanth/anaconda/envs/rlpe/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/hemanth/anaconda/envs/rlpe/lib/python3.12/site-packages/transformers/models/vit/modeling_vit.py", line 183, in forward
    embeddings = self.projection(pixel_values).flatten(2).transpose(1, 2)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/hemanth/anaconda/envs/rlpe/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/hemanth/anaconda/envs/rlpe/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/hemanth/anaconda/envs/rlpe/lib/python3.12/site-packages/torch/nn/modules/conv.py", line 554, in forward
    return self._conv_forward(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/hemanth/anaconda/envs/rlpe/lib/python3.12/site-packages/torch/nn/modules/conv.py", line 549, in _conv_forward
    return F.conv2d(
           ^^^^^^^^^
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor

from torchvision import transforms
import torch
import torch.optim as optim
from torch import nn
from art.estimators.classification import PyTorchClassifier

from rldata import (
DatasetConfig,
datapipeline,
)
from model import ImageClassificationPipeline,ModelSource
from attacks import apply_attack,LIST_OF_ATTACTS
from defences import DefenseManager
from metrics import MetricsCalculator,print_metrics

#=====================Data====================
data_config=DatasetConfig(
    dataset_path="zh-plus/tiny-imagenet",
    split_config={"train": 0.8, "test": 0.2},
    columns=["image","label"],
    num_samples=1000,
    seed=42,
    kwargs={"cache_dir": "./data",}
)

transforms_fnc=transforms.Compose([
    transforms.Lambda(lambda img: img.convert("RGB")),  # Convert to RGB first
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],  # Use three values for three channels
                       std=[0.229, 0.224, 0.225])
])

#=====================Data processing====================
train_loader, test_loader= datapipeline(
    config=data_config,
    transforms_fn=transforms_fnc,
    target_column='image',
    label_column='label',
    
)

#=====================Model====================



import torch
import torch.nn as nn
from transformers import ViTForImageClassification

class WrappedViTModel(nn.Module):
    def __init__(self, original_model):
        super(WrappedViTModel, self).__init__()
        self.original_model = original_model

    def forward(self, x):
        # Obtain the logits from the HuggingFace model's output
        output = self.original_model(x)
        return output.logits
from transformers import ViTImageProcessorFast

# Initialize the pipeline
model_pipeline = ImageClassificationPipeline(
    model_name_or_path="google/vit-base-patch16-224-in21k",
    model_source=ModelSource.TRANSFORMERS.value,
    cache_dir="./data",
)

# Extract the model from the pipeline
original_model = model_pipeline.model

# Modify the classifier as you've done
original_model.classifier = torch.nn.Linear(
    in_features=original_model.classifier.in_features,
    out_features=1000,  # Ensure this matches your number of classes
    bias=True
)

# Wrap the model
wrapped_model = WrappedViTModel(original_model)

# Move the model to the appropriate device
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
wrapped_model.to(device)
wrapped_model.eval()  # Set to evaluation mode



from art.estimators.classification import PyTorchClassifier
import torch.optim as optim
import torch.nn as nn

classifier = PyTorchClassifier(
    model=wrapped_model,
    loss=nn.CrossEntropyLoss(),
    optimizer=optim.Adam(wrapped_model.parameters()),
    input_shape=(3, 224, 224),
    nb_classes=1000,
    clip_values=(0.0, 1.0),  # Ensure these values match your data preprocessing
    device_type="cuda" if torch.cuda.is_available() else "cpu"
)

for batch in train_loader:
    images, labels = batch["images"].to(device), batch["labels"].to(device)
    defence_generator = DefenseManager()
    list_of_defences = defence_generator.list_defenses()
    print(list_of_defences)

    for attack in LIST_OF_ATTACTS:
        images = apply_attack(attack_name=attack, classifier=classifier, x=images, y=labels)
        model_output = wrapped_model(images)
        metrics_pipeline = MetricsCalculator(preds=model_output, labels=labels, num_classes=1000)
        metrics = metrics_pipeline.get_all_metrics()
        print_metrics(metrics)

    exit()
api details



def apply_attack(
    attack_name: str,
    classifier: Any,
    x: np.ndarray,
    y: Optional[np.ndarray] = None,
    new_params: Optional[Dict[str, Any]] = None,
) -> np.ndarray:
    """
    Apply a single adversarial attack with the ability to update attack parameters.
    """
    logger.info(f"Initializing attack: {attack_name}")
    try:
        attack_class = get_attack_class(attack_name)
        if not attack_class:
            raise NotImplementedError(f"The attack '{attack_name}' is not supported.")

        params = get_attack_parameters(attack_name)

        if new_params:
            params = update_attack_parameters(params, new_params)

        # Correctly assign the classifier or estimator
        if "estimator" in params:
            params["estimator"] = classifier
        if "classifier" in params:
            params["classifier"] = classifier

        # Handle special parameters if necessary
        if attack_name == "AutoAttack":
            params["estimator_orig"] = classifier
        elif attack_name == "AdversarialPatchPyTorch":
            params["estimator"] = classifier

        # Initialize the attack with the classifier and parameters
        attack = attack_class(**params)
        
        # Ensure inputs are on CPU and in numpy format
        if isinstance(x, torch.Tensor):
            x = x.detach().cpu()
        if isinstance(y, torch.Tensor):
            y = y.detach().cpu()
            
        # Convert to numpy arrays
        x_numpy = x.numpy() if isinstance(x, torch.Tensor) else x
        y_numpy = y.numpy() if isinstance(y, torch.Tensor) else y
        
        # Ensure labels are in the correct format (integers)
        if y_numpy is not None:
            y_numpy = y_numpy.astype(np.int64)
            
        try:
            # Generate adversarial examples
            adversarial_examples = attack.generate(x=x_numpy, y=y_numpy)
            
            # Convert back to tensor if input was tensor
            if isinstance(x, torch.Tensor):
                adversarial_examples = torch.from_numpy(adversarial_examples).to(x.device)
                
            logger.info(f"Attack '{attack_name}' performed successfully.")
            return adversarial_examples
            
        except Exception as e:
            logger.error(f"Attack generation failed: {str(e)}")
            raise

    except ValueError as ve:
        logger.error(f"Value error: {ve}")
        raise
    except TypeError as te:
        logger.error(
            f"Type error while initializing '{attack_name}': {te}. Check parameter compatibility."
        )
        raise
    except NotImplementedError as nie:
        logger.warning(str(nie))
        raise
    except Exception as e:
        logger.error(f"Unexpected error during '{attack_name}': {e}")
        raise


LIST_OF_ATTACTS: List[str] = [
        'CarliniL2Method',
        'CarliniLInfMethod',
        'DeepFool',
        'FastGradientMethod',
        'BasicIterativeMethod',
        'PixelAttack',
        'ProjectedGradientDescent',
        'SpatialTransformation',
        'SquareAttack',
        'UniversalPerturbation',
        'ZooAttack',
    ]