Package: transformer-pytorch
Transformer-pytorchA PyTorch implementation of Transformer from "Attention is All You Need" (https://arxiv.org/abs/1706.03762).This repo focuses on clean, readable, and modular implementation of the paper.$ docker build --tag transformer --rm .
$ docker run --name $CONTAINER_NAME -it --gpus all --shm-size 16G --volume $(pwd):/transformer transformer bashpython main.py v1 prepare multi30k data/Multi30k/python -m spacy download en
python -m spacy download depython main.py v1 build-vocab data/Multi30k/train.json results/vocabs/shared_vocab.tsv --source-language de --target-language en --min-freq 3Vocabulary 9521 lines written to results/vocabs/shared_vocab.tsvCUDA_VISIBLE_DEVICES=0 python main.py v1 train data/Multi30k/train.json data/Multi30k/val.json results/vocabs/shared_vocab.tsv results/vocabs/shared_vocab.tsv results/runs playgroundpython main.py v1 prepare aihub data/AIHub/RequirementsPython 3.6+PyTorch 4.1+NumPyNLTKtqdmUsagePrepare datasetsThis repo comes with example data indata/directory. To begin, you will need to prepare datasets with given data as follows:$ python prepare_datasets.py --train_source=data/example/raw/src-train.txt --train_target=data/example/raw/tgt-train.txt --val_source=data/example/raw/src-val.txt --val_target=data/example/raw/tgt-val.txt --save_data_dir=data/example/processedThe example data is brought fromOpenNMT-py.
The data consists of parallel source (src) and target (tgt) data for training and validation.
A data file contains one sentence per line with tokens separated by a space.
Below are the provided example data files.src-train.txttgt-train.txtsrc-val.txttgt-val.txtTrain modelTo train model, provide the train script with a path to processed data and save files as follows:$ python train.py --data_dir=data/example/processed --save_config=checkpoints/example_config.json --save_checkpoint=checkpoints/example_model.pth --save_log=logs/example.logThis saves model config and checkpoints to given files, respectively.
You can play around with hyperparameters of the model with command line arguments.
For example, add--epochs=300to set the number of epochs to 300.TranslateTo translate a sentence in source language to target language:$ python predict.py --source="There is an imbalance here ." --config=checkpoints/example_config.json --checkpoint=checkpoints/example_model.pth

Candidate 0 : Hier fehlt das Gleichgewicht .
Candidate 1 : Hier fehlt das das Gleichgewicht .
Candidate 2 : Hier fehlt das das das Gleichgewicht .It will give you translation candidates of the given source sentence.
You can adjust the number of candidates with command line argument.EvaluateTo calculate BLEU score of a trained model:$ python evaluate.py --save_result=logs/example_eval.txt --config=checkpoints/example_config.json --checkpoint=checkpoints/example_model.pth

BLEU score : 0.0007947File descriptionmodels.pyincludes Transformer's encoder, decoder, and multi-head attention.embeddings.pycontains positional encoding.losses.pycontains label smoothing loss.optimizers.pycontains Noam optimizer.metrics.pycontains accuracy metric.beam.pycontains beam search.datasets.pyhas code for loading and processing data.trainer.pyhas code for training model.prepare_datasets.pyprocesses data.train.pytrains model.predict.pytranslates given source sentence with a trained model.evaluate.pycalculates BLEU score of a trained model.ReferenceOpenNMT-pyAuthor@dreamgonflyDeploypython3 setup.py sdist bdist_wheelpython3 -m twine upload --repository testpypi dist/*
python3 -m twine upload --repository dist/*