Kspsvln/INLP_A3

INLP Assignment 3: Checkpoint Collection Model Card

This repository contains PyTorch checkpoints for a multi-stage NLP system built for noisy cipher text recovery:

1. Task 1: character-level cipher-to-plain decryption (RNN, LSTM)
2. Task 2: plain-text language modeling (BiLSTM MLM, SSM NWP)
3. Task 3: correction pipeline combining Task 1 + Task 2 models

Model Summary

This is a checkpoint collection, not a single Transformers model. Inference uses custom project code (main.py, src/task1/*, src/task2/*, src/task3/*).

Primary checkpoints currently used by configs:

• checkpoints/task1/rnn_task4_complex_best.pt
• checkpoints/task1/lstm_task4_complex_best.pt
• checkpoints/task2/bilstm_complex_best.pt
• checkpoints/task2/ssm_complex_best.pt

Associated vocab files:

• checkpoints/task1/rnn_vocab.json
• checkpoints/task1/lstm_vocab.json
• checkpoints/task2/bilstm_vocabv2.json
• checkpoints/task2/ssm_vocabv2.json

Additional historical checkpoints (epoch snapshots and earlier best versions) are also included under checkpoints/task1/, checkpoints/task2/, and checkpoints/task3/.

Intended Use

• Research/assignment evaluation for cipher decryption and LM-assisted correction.
• Comparative experiments across RNN/LSTM/SSM/BiLSTM variants.

Not intended for:

• Production security/cryptanalysis systems.
• General-domain text generation quality tasks.

Training Data

• Plain text: data/plain.txt
• Cipher text variants: data/cipher_00.txt to data/cipher_04.txt

The models are trained and evaluated within this assignment dataset setup.

Configuration (Current Complex Setup)

Task 1 RNN Decryption

• Config: config/task1/rnn.yaml
• Architecture: embed=256, hidden=512, layers=2, dropout=0.30
• Sequence length: 50
• Training: epochs=50, batch_size=128, lr=1e-3, grad_clip=1.0
• Checkpoint: rnn_task4_complex_best.pt

Task 1 LSTM Decryption

• Config: config/task1/lstm.yaml
• Architecture: embed=256, hidden=512, layers=2, dropout=0.30
• Sequence length: 50
• Training: epochs=60, batch_size=128, lr=1e-3, grad_clip=1.0
• Checkpoint: lstm_task4_complex_best.pt

Task 2 BiLSTM MLM

• Config: config/task2/bilstm.yaml
• Architecture: embed=128, hidden=128, layers=2, dropout=0.15, max_seq_len=24
• Training: epochs=45, batch_size=48, lr=1e-4, weight_decay=1e-5
• Checkpoint: bilstm_complex_best.pt

Task 2 SSM NWP

• Config: config/task2/ssm.yaml
• Architecture: embed=64, d_state=32, layers=1, dropout=0.50, max_seq_len=24
• Training: epochs=50, batch_size=16, lr=6e-4, weight_decay=2e-2
• Checkpoint: ssm_complex_best.pt

Evaluation Results

Task 1 (from outputs/task1_rnn.txt, outputs/task1_lstm.txt)

Model	Loss	Accuracy	Perplexity	Character Accuracy	Word Accuracy	Levenshtein Distance
RNN	1.495032	0.579503	4.459479	0.579503	0.390902	20.781357
LSTM	1.331345	0.631559	3.786133	0.580139	0.433234	19.153613

Task 2 (from outputs/task2_bilstm.txt, outputs/task2_ssm.txt)

Model	Validation Loss	Perplexity
BiLSTM MLM	5.5569	259.01
SSM NWP	6.0509	424.50

Task 3 (from outputs/task3_*.csv, noise levels cipher_01..cipher_04)

Average metrics:

• Decryption only: char_acc=0.4393, word_acc=0.0161, norm_edit_dist=0.5298, bleu=0.0161, rougeL=0.1904
• BiLSTM correction: char_acc=0.0814, word_acc=0.0165, norm_edit_dist=0.5731, bleu=0.0147, rougeL=0.1925
• SSM correction: char_acc=0.0801, word_acc=0.0169, norm_edit_dist=0.5594, bleu=0.0147, rougeL=0.1786

How to Use

This repo uses a custom CLI (uv + main.py) rather than the Transformers pipeline API.

Setup

uv sync

Evaluate checkpoints

uv run main.py task1_rnn --mode evaluate --config config/task1/rnn.yaml
uv run main.py task1_lstm --mode evaluate --config config/task1/lstm.yaml
uv run main.py task2_bilstm --mode evaluate --config config/task2/bilstm.yaml
uv run main.py task2_ssm --mode evaluate --config config/task2/ssm.yaml
uv run main.py task3_bilstm --mode evaluate --config config/task3/bilstm.yaml
uv run main.py task3_ssm --mode evaluate --config config/task3/ssm.yaml

Hugging Face loading behavior in this project

The code attempts Hugging Face model download first when huggingface.repo_id and filenames are configured, then falls back to local checkpoints.

Limitations

• Domain-specific dataset; results may not generalize.
• Task 3 correction models can reduce character-level fidelity in current setup.
• Models depend on exact vocab/checkpoint pairing.
• Not packaged as a single AutoModel for generic HF inference.

Ethical Considerations

• This work is for educational experimentation.
• No guarantee for security-critical decryption decisions.
• Human review is recommended for downstream interpretation.

Reproducibility Notes

• Use YAML configs in config/task1/, config/task2/, config/task3/.
• Keep checkpoint filenames and vocab files aligned.
• Runtime and outputs may vary by hardware (CPU/GPU) and seed settings.

Citation

If you use this work, cite the repository and assignment context.

Contact

Owner: Kspsvln/INLP_A3