LayerNorm/attnResNet-50

AttnRes-50: 2D Attention Residuals for ResNet-50

Model Description

Try to extends the original 1D AttnRes (https://arxiv.org/abs/2603.15031) to 2D spatial feature maps for computer vision. However it seems not work on Classification :(

Mode	Description
none	Standard ResNet-50 (baseline)
stage	Intra-stage AttnRes — each block attends to prior block outputs within the same stage
global	Cross-stage AttnRes — each block attends to all prior block outputs across stages, with spatial/channel alignment via interpolation and 1×1 conv projections

Dataset: ImageNet-100

ImageNet-100 is a 100-class subset of ImageNet-1K (ILSVRC2012). This study uses the split defined by Tian et al. (2020).

Split	Samples
Total	126,689
Training (80%)	101,351
Validation (20%)	25,338

The train/val split is deterministic (torch.manual_seed(42) with randperm).

Training Details

Hyperparameters

Parameter	Value
Optimizer	SGD with momentum (µ=0.9)
Learning rate	0.1 (cosine annealing to 0 over 90 epochs)
Weight decay	1×10⁻⁴
Batch size	32
Epochs	90
Input size	224×224
GPU	NVIDIA A100 (80GB) — single GPU

Data Augmentation

• Training: RandomResizedCrop(224), RandomHorizontalFlip, ToTensor, Normalize(mean=[0.485,0.456,0.406], std=[0.229,0.224,0.225])
• Validation: Resize(256), CenterCrop(224), ToTensor, Normalize (same stats)

Results

Mode	Acc@1	Acc@5	Val Loss	Macro F1	Macro AUC	Params
none (Baseline)	86.86%	96.83%	0.5034	86.65%	99.75%	23.77M
stage (Intra-Stage)	85.80%	96.62%	0.5268	85.59%	99.74%	23.77M
global (Cross-Stage)	85.45%	96.57%	0.5380	85.23%	99.72%	28.36M

• Both attention variants underperform the standard ResNet-50 baseline on ImageNet-100 (~1–1.4% lower Acc@1).

Usage

import torch
from attnres_resnet_demo import AttnResNet50

# Load baseline (standard ResNet-50)
model = AttnResNet50(mode='none', num_classes=100)
ckpt = torch.load('result/best_none.pth', map_location='cpu')
model.load_state_dict(ckpt['model_state_dict'])
model.eval()

# Inference
from PIL import Image
import torchvision.transforms as T

transform = T.Compose([
    T.Resize(256),
    T.CenterCrop(224),
    T.ToTensor(),
    T.Normalize(mean=[0.485, 0.456, 0.406],
                std=[0.229, 0.224, 0.225]),
])

img = Image.open('example.jpg')
x = transform(img).unsqueeze(0)
with torch.no_grad():
    logits = model(x)
    pred = logits.argmax(dim=1)

Checkpoints

Mode	File	Size
none	best_none.pth	182 MB
stage	best_stage.pth	182 MB
global	best_global.pth	200 MB

Each checkpoint contains:

{
    'epoch': int,
    'mode': str,            # 'none' | 'stage' | 'global'
    'model_state_dict': OrderedDict,
    'optimizer_state_dict': OrderedDict,
    'best_acc1': float,
    'num_classes': int,      # 100
}

Note for global mode: The model uses a lazy-initialized ModuleDict for cross-stage channel projections. When loading the checkpoint, run a dummy forward pass first to populate the projection layers before calling load_state_dict.

Limitations

• Small-scale study: ImageNet-100 is only 100 classes; conclusions may not transfer to full ImageNet-1K.
• No dual mode results: The dual-path variant was not trained due to negative findings from stage and global.
• Single training run per mode: No repeated runs with different seeds to measure variance.
• Standard training recipe: Uses a basic SGD + cosine annealing schedule without modern tricks (warmup, label smoothing, mixup, EMA).

Citation

If you use this code or findings, please cite the original AttnRes work:

@misc{chen2026attnres,
  title         = {Attention Residuals},
  author        = {Kimi Team  and Chen, Guangyu  and Zhang, Yu  and Su, Jianlin  and Xu, Weixin  and Pan, Siyuan  and Wang, Yaoyu  and Wang, Yucheng  and Chen, Guanduo  and Yin, Bohong  and Chen, Yutian  and Yan, Junjie  and Wei, Ming  and Zhang, Y.  and Meng, Fanqing  and Hong, Chao  and Xie, Xiaotong  and Liu, Shaowei  and Lu, Enzhe  and Tai, Yunpeng  and Chen, Yanru  and Men, Xin  and Guo, Haiqing  and Charles, Y.  and Lu, Haoyu  and Sui, Lin  and Zhu, Jinguo  and Zhou, Zaida  and He, Weiran  and Huang, Weixiao  and Xu, Xinran  and Wang, Yuzhi  and Lai, Guokun  and Du, Yulun  and Wu, Yuxin  and Yang, Zhilin  and Zhou, Xinyu},
  year          = {2026},
  archiveprefix = {arXiv},
  eprint        = {2603.15031},
  primaryclass  = {cs.CL}
}