One Activation for the Whole Network: Five Learnable Fourier Coefficients Outperform GELU on ImageNet-1K
FAct replaces every per-layer, fixed activation function in a network with ONE shared, learnable activation — parameterized by just 5 scalars — that is optimized jointly with the network's weights by ordinary backprop. No per-layer copies, no architecture search, no separate meta-learning loop: the activation is just 5 more entries in the same parameter vector AdamW already updates every step.
Concretely, the activation is a truncated Fourier series with K=2
harmonics,
phi(t) = a0 + a1*cos(w t) + b1*sin(w t) + a2*cos(2w t) + b2*sin(2w t)
with {a0, a1, b1, a2, b2} — 5 learnable scalars, tied across every
neuron in every layer of the network — initialized to exactly reproduce
GELU's own Fourier coefficients, so training starts identical to a GELU
baseline and is then free to deviate. This repo is the reproducibility
package for the first empirical test of that idea: does letting the network
co-adapt its one global activation shape, at a +5-parameter cost, beat a
fixed-GELU baseline of otherwise identical size and training recipe?
Everything needed to reproduce the numbers below — training code, exact
launch commands, best/final checkpoints, per-epoch metric logs, plots, and
the scripts used to regenerate them — is in this repo. (In the code, and in
every checkpoint's variant field, this mechanism is called
fact_k2_global.)
Naming. FAct — Fourier Approximation for the activation
(function): rather than picking an activation by hand (ReLU, GELU, ...),
approximate it as a truncated Fourier series and let training choose its
coefficients. The broader FAct study spans several variants of that idea —
per-neuron, per-layer, and network-wide coefficient sharing, at various
harmonic counts K — evaluated across Fashion-MNIST/CIFAR-10/CIFAR-100/
Tiny-ImageNet/ImageNet-1K and, for one variant, WMT14 NMT. This repo
documents its headline configuration: K=2, coefficients shared across
the entire network (not just within a layer), so the whole model's
nonlinearity is jointly optimized through exactly 5 scalars. Throughout this
card, "FAct" means that configuration — fact_k2_global in the code — and
the baseline it is compared against is a fixed per-layer GELU.
Results: ImageNet-1K, 100 epochs, 5 seeds
ViT (depth=6, embed_dim=192, heads=6, mlp_ratio=4.0, patch16/224). Baseline has 3,048,232 trainable params (fixed GELU, one per layer); FAct has 3,048,237 — exactly +5 params for the one shared, learnable activation, network-wide.
| Variant | seed | best val_acc | test top-1 (best ckpt) | test top-5 |
|---|---|---|---|---|
| baseline (fixed GELU) | 1 | 0.6518 | 0.6250 | 0.8427 |
| baseline (fixed GELU) | 2 | 0.6478 | 0.6256 | 0.8424 |
| baseline (fixed GELU) | 3 | 0.6523 | 0.6249 | 0.8400 |
| baseline (fixed GELU) | 4 | 0.6490 | 0.6223 | 0.8400 |
| baseline (fixed GELU) | 5 | 0.6479 | 0.6249 | 0.8421 |
| baseline mean ± std (n=5) | 0.6498 ± 0.0019 | 0.6245 ± 0.0012 | 0.8414 ± 0.0012 | |
| FAct (5 learnable params) | 1 | 0.6768 | 0.6512 | 0.8602 |
| FAct (5 learnable params) | 2 | 0.6734 | 0.6433 | 0.8556 |
| FAct (5 learnable params) | 3 | 0.6643 | 0.6396 | 0.8527 |
| FAct (5 learnable params) | 4 | 0.6691 | 0.6424 | 0.8547 |
| FAct (5 learnable params) | 5 | 0.6681 | 0.6427 | 0.8562 |
| FAct mean ± std (n=5) | 0.6703 ± 0.0043 | 0.6438 ± 0.0039 | 0.8559 ± 0.0024 |
+5 parameters (a 0.00016% size increase) buys +2.1pt val_acc / +1.9pt test top-1 / +1.4pt test top-5 on average across all 5 seeds — jointly optimizing the activation's shape alongside the weights outperforms fixing it to GELU throughout training, at a parameter cost that rounds to zero. The effect is consistent across every individual seed (baseline never comes within 1pt of FAct on any seed, top-1 or top-5, 5/5).
Area under the accuracy curve (trapezoidal integral over epoch,
normalized to the same [0, 1] scale as accuracy — i.e. the average accuracy
sustained across the whole 100-epoch run, not just the final value; see
code/compute_auc.py / results/auc_summary.json) tells the same story
from a different angle — FAct isn't just ending higher, it's higher
throughout training:
| Variant | AUC(train_acc) | AUC(val_acc) |
|---|---|---|
| baseline mean ± std (n=5) | 0.5295 ± 0.0011 | 0.5384 ± 0.0010 |
| FAct mean ± std (n=5) | 0.5539 ± 0.0029 | 0.5592 ± 0.0036 |
Convergence speed. The sharper form of the same question — not "how high
on average" but "how many epochs to get there" (code/convergence_milestones.py
/ results/convergence_milestones.json). Mean ± std of the first epoch whose
val_acc reaches each threshold; (n/5) is how many seeds reach it at all
within the 100-epoch budget:
| val_acc threshold | baseline (fixed GELU) | FAct | epochs saved |
|---|---|---|---|
| 50% | 27.6 ± 1.0 (5/5) | 22.2 ± 1.7 (5/5) | 5.4 |
| 55% | 49.6 ± 1.6 (5/5) | 42.4 ± 2.3 (5/5) | 7.2 |
| 60% | 70.8 ± 1.3 (5/5) | 61.4 ± 1.9 (5/5) | 9.4 |
| 65% | 98.5 ± 1.5 (2/5) | 82.8 ± 1.6 (5/5) | 16.5 (n=2) |
Read the n column before the means: the gap widens the higher the bar, and
at 65% it stops being a speed difference at all — all five FAct seeds reach
65% val_acc, at epoch 82.8 on average, while three of the five baseline seeds
never reach it within 100 epochs (the two that do arrive at epoch ~98.5, so
the "epochs saved" figure on that row averages only those two pairs).
FAct's validation loss is also below its seed-matched baseline's from epoch 1
onward, in all 5 seeds — the lead is present from the start, not acquired
late. The flip side is a wider generalisation gap at epoch 100: train_acc
minus val_acc is 4.24 ± 0.19 pp for FAct versus 3.11 ± 0.26 pp for the
baseline.
See figures/imagenet1k_100ep_accuracy_curves.png and ..._loss_curves.png
for the all-seeds/mean view, and ..._seed1_* through ..._seed5_* for the
per-seed loss/accuracy curves (10 figures total).
Compute cost: FLOPs, inference speed, training speed
FLOPs are identical between variants. code/benchmark_speed_flops.py
counts multiply-accumulate FLOPs (torch.utils.flop_counter.FlopCounterMode)
for one forward pass and one forward+backward pass, per image:
| forward | forward + backward | |
|---|---|---|
| both variants (per image, 224×224) | 1.283 GFLOPs | 3.791 GFLOPs |
Identical because standard FLOP counters only tally matmul/conv ops — the activation itself (GELU's erf, or FAct's 4 cos/sin evaluations) is elementwise and not counted, on either side. But wall-clock time is not identical — FAct's Fourier terms are real, if cheap, transcendental-function kernels launched at every activation site, and that shows up in practice despite the FLOP count being a tie:
| standard (GELU) | FAct (5 params) | overhead | |
|---|---|---|---|
| inference, batch=1 | 1.80 ms/img (556 img/s) | 2.98–3.35 ms/img (299–336 img/s) | ~1.7–1.9x |
| inference, batch=256 | 0.035 ms/img (28.6K img/s) | 0.090 ms/img (11.1K img/s) | ~2.6x |
| inference, batch=1024 | 0.034 ms/img (29.6K img/s) | 0.087 ms/img (11.5K img/s) | ~2.6x |
| training step, batch=256 (fwd+bwd+AdamW) | 54–55 ms/step (4.6–4.7K img/s) | 71–72 ms/step (3.5–3.6K img/s) | ~1.3x |
| est. training time / epoch (1.27M train images) | ~4.5–4.6 min | ~5.9–6.0 min | ~1.3x |
Measured back-to-back on the same, currently-free GPU (NVIDIA H200 NVL,
architecture-only — randomly-initialized weights, since weights don't affect
speed/FLOPs), 10 warmup + 50 timed iters per config, batch=256 for the
training step (matches the actual training recipe). Run
python code/benchmark_speed_flops.py to reproduce.
Repo structure
.
├── README.md
├── requirements.txt
├── code/ # everything needed to train/eval/reproduce
│ ├── cv_train.py # training loop (AdamW, cosine LR, AMP, label smoothing)
│ ├── cv_vit.py # ViT definition (build_vit); builds the ONE shared activation
│ ├── cv_data.py # ImageNet-1K dataloaders (parquet-backed)
│ ├── fourier_ffn.py # the two FFNs (fixed-GELU vs shared FAct) + name -> FFN dispatch
│ ├── fourier_layers.py # FourierActivation (the 5-param FAct module) + coeff utilities
│ ├── eval_checkpoint.py # re-evaluate any checkpoint on val/test (top-1 only)
│ ├── eval_top5.py # + top-1 AND top-5 on the official test split, all 10 best.pt
│ ├── compute_auc.py # area under the train/val accuracy curve, per run + mean±std
│ ├── convergence_milestones.py # epochs to reach each val_acc threshold, per run + mean±std
│ ├── benchmark_speed_flops.py # FLOPs + inference/training-step speed, both variants, same GPU
│ ├── download_imagenet1k.py # pull ILSVRC/imagenet-1k parquet shards from HF
│ ├── extract_imagenet1k_coeffs.py # dump a FAct checkpoint's learned {a0,a1,b1,a2,b2}
│ ├── make_results_json.py # logs/*.log -> results/cv_imagenet1k_100ep.json
│ ├── plot_curves.py # results/*.json -> figures/*_loss|accuracy_curves.png (all seeds + mean)
│ ├── plot_seed_curves.py # per-seed loss/accuracy curves (--seed N, or all five)
│ └── scripts/ # exact shell launch commands used for every run/resume
├── checkpoints/
│ ├── standard/seed{1,2,3,4,5}/{best,final}.pt # fixed-GELU baseline
│ └── fact_k2_global/seed{1,2,3,4,5}/{best,final}.pt # FAct -- all 5 seeds complete
├── results/
│ ├── cv_imagenet1k_100ep.json # per-epoch train/val loss+acc history for all 10 runs
│ ├── top5_eval.json # top-1 (sanity-check) + top-5 test acc, all 10 best.pt (code/eval_top5.py)
│ ├── auc_summary.json # area-under-accuracy-curve, per run + mean±std (code/compute_auc.py)
│ ├── convergence_milestones.json # epochs-to-threshold, final/best val_acc, val-loss lead,
│ │ # train/val gap (code/convergence_milestones.py)
│ ├── speed_flops_benchmark.json # FLOPs + inference/training speed, both variants, same GPU
│ │ # (code/benchmark_speed_flops.py)
│ ├── fact_k2_global_coeffs_seed{4,5}.json # FAct's learned {a0,a1,b1,a2,b2}, init vs. final,
│ │ # per seed (code/extract_imagenet1k_coeffs.py; run it
│ │ # with --seed 1/2/3 to generate the other three)
│ └── fact_k2_global_coeffs_seed0_pilot.json # same, from an earlier seed-0 pilot run,
│ # not one of the 5 seeds above
├── figures/
│ ├── imagenet1k_100ep_loss_curves.png # all seeds, mean per variant
│ ├── imagenet1k_100ep_accuracy_curves.png # all seeds, mean per variant
│ ├── imagenet1k_100ep_seed1_loss_curves.png # seed-1-only view
│ ├── imagenet1k_100ep_seed1_accuracy_curves.png # seed-1-only view
│ ├── imagenet1k_100ep_seed2_loss_curves.png # seed-2-only view
│ ├── imagenet1k_100ep_seed2_accuracy_curves.png # seed-2-only view
│ ├── imagenet1k_100ep_seed3_loss_curves.png # seed-3-only view
│ ├── imagenet1k_100ep_seed3_accuracy_curves.png # seed-3-only view
│ ├── imagenet1k_100ep_seed4_loss_curves.png # seed-4-only view
│ ├── imagenet1k_100ep_seed4_accuracy_curves.png # seed-4-only view
│ ├── imagenet1k_100ep_seed5_loss_curves.png # seed-5-only view
│ └── imagenet1k_100ep_seed5_accuracy_curves.png # seed-5-only view
└── logs/ # raw stdout/stderr training logs (source of truth for
# results/cv_imagenet1k_100ep.json)
Reproducing
pip install -r requirements.txt
# 1. Get the data (ImageNet-1K train+val parquet shards, ~145GB)
python code/download_imagenet1k.py # edit LOCAL_DIR at the top first, or pass --local-dir
# 2. Train (seed 1, fixed-GELU baseline; ~15-17h on a single modern GPU)
python code/cv_train.py --dataset imagenet1k --variant standard --act-ref gelu \
--seed 1 --epochs 100 --save-epoch-ckpts --out-dir <out_dir>
# 2'. Train (seed 1, FAct -- 5 jointly-optimized activation params)
python code/cv_train.py --dataset imagenet1k --variant fact_k2_global \
--seed 1 --epochs 100 --save-epoch-ckpts --out-dir <out_dir>
# See code/scripts/*.sh for the exact commands (incl. --resume-from) used
# for every seed of this run, including mid-training resumes after
# unexpected host churn (seed1 FAct @ epoch 47, seed3 both variants @
# epoch 51/43, seed5 standard @ epoch 42) -- the cosine LR schedule stays
# continuous across a resume since --epochs after --resume-from means
# "additional epochs" and schedule_epochs is offset accordingly (see
# cv_train.py's __main__). seed4 (both variants) and seed5's fact_k2_global
# leg trained end-to-end without a resume, on different hosts.
# 3. Evaluate any checkpoint (top-1 only)
python code/eval_checkpoint.py --checkpoint checkpoints/fact_k2_global/seed1/best.pt \
--dataset imagenet1k --variant fact_k2_global
# 3'. Top-1 + top-5 on the official test split, for all 10 best.pt at once
python code/eval_top5.py
# 3''. Area under the train/val accuracy curve, per run + mean±std
python code/compute_auc.py
# 3'''. Epochs to reach each val_acc threshold, per run + mean±std
python code/convergence_milestones.py
# 3''''. FLOPs + inference/training-step speed, both variants, current GPU
python code/benchmark_speed_flops.py
# 4. Inspect FAct's 5 learned coefficients vs. its GELU initialization
python code/extract_imagenet1k_coeffs.py # seed 1 by default; --seed 2/3/4/5
# 5. Regenerate results.json + plots from the raw logs
python code/make_results_json.py
python code/plot_curves.py # all-seeds + mean figures
python code/plot_seed_curves.py # per-seed figures (add --seed N for just one)
How the 5 parameters are optimized
There is nothing bilevel or alternating about it: {a0, a1, b1, a2, b2} are
registered as ordinary nn.Parameters on one FourierActivation module,
constructed once in cv_vit.py and passed by reference into every
transformer block's FFN (fourier_ffn.py), so all layers share the exact
same 5 numbers at every forward pass. They receive gradients through the
same loss.backward() call as every weight matrix and get the same AdamW
update in the same opt.step() — the only special-casing is at
initialization, where {a0,a1,b1,a2,b2} are set to GELU's true Fourier
coefficients (fourier_layers.true_fourier_coeffs) rather than a random
init, so training starts at parity with the baseline and any divergence is
attributable to what the optimizer does with those 5 extra degrees of
freedom.
Training config
AdamW, lr=1e-3, weight_decay=0.05, 5-epoch linear warmup then cosine decay
to 1% of peak, batch_size=256, label_smoothing=0.1, dropout=0.1, standard
random-crop+flip augmentation, AMP mixed precision. Identical recipe across
both variants and all seeds — the only difference is fixed-per-layer GELU
(baseline) vs. one shared, jointly-optimized 5-parameter FAct activation
(GELU-initialized). Full config is recorded per-run in
results/cv_imagenet1k_100ep.json and in each checkpoint's config field.
Notes
- Checkpoints:
best.pt(best val-accuracy epoch) andfinal.pt(last epoch) are included per run, sufficient to reproduce the reported numbers and to evaluate. Full per-epoch snapshots are also included undercheckpoints/<variant>/seed<N>/epochs/epoch<NNN>.pt(~1.2GB/run) -- needed for--resume-from, and for regenerating any of this repo's own training curves from scratch rather than fromresults/*.json. ILSVRC/imagenet-1kon the Hub has its own (research, non-commercial) usage terms — this repo doesn't redistribute the dataset itself, only weights trained on it and metrics computed from it.- The
mitlicense tag above covers the code in this repo; it does not override ImageNet-1K's own terms for the data or (arguably) weights trained on it.