Linear-Transformer
Creators: name: Juncai Guo
Publisher: name: Zenodo
Year: 2021

description: 1. Runtime Environment:<br> 4 NVIDIA 2080 Ti GPUs<br> Ubuntu 16.04<br> CUDA 10.0 (with CuDNN of the corresponding version)<br> Python 3.7<br> PyTorch 1.2.0<br> 2. Before the test, you should parse the GLOVE embeddings: <br> 1) Download the embedding data from <br> http://downloads.cs.stanford.edu/nlp/data/glove.840B.300d.zip <br> to the directory <br> “ ./exp_linear_transformer/data/glove/ ”<br> 2) unzip the zip file with the name “ glove.840B.300d.txt ”<br> 3) run the python file <br> “ parse_glove.py ” in the directory “ ./exp_linear_transformer/src_code/parse_glove/ “<br> 3. Then, install the fastnlp 0.4.1 with pip. If other packages are missed, install them with pip.<br> 4. Details about model tests are as follows:<br> 1) Ex1. Sentiment Classification on SST dataset<br> a. Download the dataset files from <br> https://nlp.stanford.edu/sentiment/trainDevTestTrees_PTB.zip and <br> https://nlp.stanford.edu/~socherr/stanfordSentimentTreebank.zip,<br> Unzip both the zip files and copy all the files<br> to the directory<br> “ ./exp_linear_transformer/data/SST/stanfordSentimentTreebank/ ”.<br> Then, go to the directory “./exp_linear_transformer/src_code/test_SST ”, and <br> b. Run the file “ make_raw_data.py ”;<br> c. Run the file “ preprocessor.py ”; <br> d. Run the file “ linear_transformer_model.py ” to test the model. The result will be printed in the end.<br> We have saved the test log into the directory " ./exp_linear_transformer/data/SST/log/ "<br> 2) Ex2. Semantic Matching on STS dataset<br> a. Download the dataset file from <br> http://ixa2.si.ehu.es/stswiki/images/4/48/Stsbenchmark.tar.gz ,<br> Unzip the file,<br> Delete the top 4 columns in the file sts-train.csv, sts-dev.csv, and sts-dev.csv,<br> Rename the 3 files as sts-train.txt, sts-dev.txt, and sts-dev.txt and copy the files to the directory<br> " ./exp_linear_transformer/data/SEMEVAL2017T1/stsbenchmark/ ".<br> Then, go to the directory “./exp_linear_transformer/src_code/test_SEMEVAL2017T1/ ”, and <br> b. Run the file “ make_raw_data.py ”;<br> c. Run the file “ preprocessor.py ”; <br> d. Run the file “ linear_transformer_model.py ”, "star_transformer_model.py", and "transformer_model.py" to test the models. The results will be printed in the end.<br> We have saved the test logs into the directory " ./exp_linear_transformer/data/SEMEVAL2017T1/log/ "<br> 3) Ex3. Language Inference on SNLI dataset<br> a. Download the dataset file from <br> https://nlp.stanford.edu/projects/snli/snli_1.0.zip<br> unzip the file and copy the unzipped files to the directory<br> " ./exp_linear_transformer/data/SNLI/snli_1.0/ ".<br> Then, go to the directory “./exp_linear_transformer/src_code/test_SNLI ", and<br> b. Run the file “ make_raw_data.py ”;<br> c. Run the file “ preprocessor.py ”; <br> d. Run the file “ linear_transformer_model.py ” to test the model. The result will be printed in the end.<br> We have saved the test log into the directory " ./exp_linear_transformer/data/SNLI/log/ "<br> 4) Ex4. Computational Analysis on STS dataset<br> a. Go to the directory “./exp_linear_transformer/src_code/test_SEMEVAL2017T1/ ”, and run the file " time_plot.py " to get the graph of " Training time v.s. sequence length ";<br> b. Reset " batch_size=16 " in the file " config.py ", run the file “ linear_transformer_model.py ”, "star_transformer_model.py", and "transformer_model.py" separately on a single GPU. According to the diffrent " Max Seq Len "s printed, record the GPU memory to the variable " length_memory " in the file " time_plot.py ". Then run the file to get the graph of " GPU memory v.s. sequence length "