SMAI Assignment 4: IOC-NN on CIFAR-10 (PyTorch, true labels)
Input-output convex network trained in Task 1.4.2 with p = 0 (no label noise) and compared with the MLP in Task 1.5.
Architecture
3072 -> 512 -> 256 -> 10, ELU in the two hidden layers, no activation on the output (10 logits, apply softmax yourself). 1,707,274 parameters. Each logit is a convex function of the input: the weights of layers 2 and 3 are non-negative.
File ioc_nn.npz: arrays W0, b0, W1, b1, W2, b2; W{i} has shape (in, out) (the transpose of the PyTorch weight), so logits = elu(elu(x @ W0 + b0) @ W1 + b1) @ W2 + b2.
Preprocessing
- Flatten each 32 x 32 x 3 image (
np.array(img).reshape(3072)). - Convert to float64 and divide by 255.
- For each of the 3072 columns subtract the mean and divide by the standard deviation of the 50,000 CIFAR-10 training images (divisor 1 if the standard deviation is 0).
Training
CIFAR-10 train part (50,000 images), cross-entropy loss, plain SGD, learning rate 0.05, batch size 64, 100 epochs, no early stopping. Layer 1 starts with the same weights and biases as the MLP; weights of layers 2 and 3 are drawn uniformly from (0, 2/n_in), biases 0. After every optimiser step each negative weight w of layers 2 and 3 is replaced by exp(w - 5). Seed 13011 (torch.manual_seed(seed + 1), DataLoader generator seed + 2).
Results (CIFAR-10 test part, 10,000 images)
Train accuracy 0.7925, test accuracy 0.4559, test loss 4.4496, mean F1 0.4479. Convexity check on 100 random pairs of test images: 0 violations.
WandB project: https://forge.coreweave.com/wandb/sushanthmantrala-iiith/SMAI-A4-MLP (run ioc-p0).