Package: skip-thoughts
Skip-Thought VectorsThis is a TensorFlow implementation of the model described in:Jamie Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel,
Antonio Torralba, Raquel Urtasun, Sanja Fidler.Skip-Thought Vectors.In NIPS, 2015.ContactCode author:Chris ShalluePull requests and issues:@cshallueContentsModel OverviewGetting StartedInstall Required PackagesDownload Pretrained Models (Optional)Training a ModelPrepare the Training DataRun the Training ScriptTrack Training ProgressExpanding the VocabularyOverviewPreparationRun the Vocabulary Expansion ScriptEvaluating a ModelOverviewPreparationRun the Evaluation TasksEncoding SentencesModel overviewTheSkip-Thoughtsmodel is a sentence encoder. It learns to encode input
sentences into a fixed-dimensional vector representation that is useful for many
tasks, for example to detect paraphrases or to classify whether a product review
is positive or negative. See theSkip-Thought Vectorspaper for details of the model architecture and more example applications.A trainedSkip-Thoughtsmodel will encode similar sentences nearby each other
in the embedding vector space. The following examples show the nearest neighbor by
cosine similarity of some sentences from themovie review dataset.Input sentenceNearest NeighborSimplistic, silly and tedious.Trite, banal, cliched, mostly inoffensive.Not so much farcical as sour.Not only unfunny, but downright repellent.A sensitive and astute first feature by Anne-Sophie Birot.Absorbing character study by André Turpin .An enthralling, entertaining feature.A slick, engrossing melodrama.Getting StartedInstall Required PackagesFirst ensure that you have installed the following required packages:Bazel(instructions)TensorFlow(instructions)NumPy(instructions)scikit-learn(instructions)Natural Language Toolkit (NLTK)First install NLTK (instructions)Then install the NLTK data (instructions)gensim(instructions)Only required if you will be expanding your vocabulary with theword2vecmodel.Download Pretrained Models (Optional)You can download model checkpoints pretrained on theBookCorpusdataset in the following
configurations:Unidirectional RNN encoder ("uni-skip" in the paper)Bidirectional RNN encoder ("bi-skip" in the paper)# Directory to download the pretrained models to.PRETRAINED_MODELS_DIR="${HOME}/skip_thoughts/pretrained/"mkdir-p${PRETRAINED_MODELS_DIR}cd${PRETRAINED_MODELS_DIR}# Download and extract the unidirectional model.wget"http://download.tensorflow.org/models/skip_thoughts_uni_2017_02_02.tar.gz"tar-xvfskip_thoughts_uni_2017_02_02.tar.gz
rmskip_thoughts_uni_2017_02_02.tar.gz# Download and extract the bidirectional model.wget"http://download.tensorflow.org/models/skip_thoughts_bi_2017_02_16.tar.gz"tar-xvfskip_thoughts_bi_2017_02_16.tar.gz
rmskip_thoughts_bi_2017_02_16.tar.gzYou can now skip to the sectionsEvaluating a ModelandEncoding Sentences.Training a ModelPrepare the Training DataTo train a model you will need to provide training data in TFRecord format. The
TFRecord format consists of a set of sharded files containing serializedtf.Exampleprotocol buffers. Eachtf.Exampleproto contains three
sentences:encode: The sentence to encode.decode_pre: The sentence precedingencodein the original text.decode_post: The sentence followingencodein the original text.Each sentence is a list of words. During preprocessing, a dictionary is created
that assigns each word in the vocabulary to an integer-valued id. Each sentence
is encoded as a list of integer word ids in thetf.Exampleprotos.We have provided a script to preprocess any set of text-files into this format.
You may wish to use theBookCorpusdataset.
Note that the preprocessing script may take12 hoursor more to complete
on this large dataset.# Comma-separated list of globs matching the input input files. The format of# the input files is assumed to be a list of newline-separated sentences, where# each sentence is already tokenized.INPUT_FILES="${HOME}/skip_thoughts/bookcorpus/*.txt"# Location to save the preprocessed training and validation data.DATA_DIR="${HOME}/skip_thoughts/data"# Build the preprocessing script.cdtensorflow-models/skip_thoughts
bazelbuild-copt//skip_thoughts/data:preprocess_dataset# Run the preprocessing script.bazel-bin/skip_thoughts/data/preprocess_dataset\--input_files=${INPUT_FILES}\--output_dir=${DATA_DIR}When the script finishes you will find 100 training files and 1 validation file
inDATA_DIR. The files will match the patternstrain-?????-of-00100andvalidation-00000-of-00001respectively.The script will also produce a file namedvocab.txt. The format of this file
is a list of newline-separated words where the word id is the corresponding 0-
based line index. Words are sorted by descending order of frequency in the input
data. Only the top 20,000 words are assigned unique ids; all other words are
assigned the "unknown id" of 1 in the processed data.Run the Training ScriptExecute the following commands to start the training script. By default it will
run for 500k steps (around 9 days on a GeForce GTX 1080 GPU).# Directory containing the preprocessed data.DATA_DIR="${HOME}/skip_thoughts/data"# Directory to save the model.MODEL_DIR="${HOME}/skip_thoughts/model"# Build the model.cdtensorflow-models/skip_thoughts
bazelbuild-copt//skip_thoughts/...# Run the training script.bazel-bin/skip_thoughts/train\--input_file_pattern="${DATA_DIR}/train-?????-of-00100"\--train_dir="${MODEL_DIR}/train"Track Training ProgressOptionally, you can run thetrack_perplexityscript in a separate process.
This will log per-word perplexity on the validation set which allows training
progress to be monitored onTensorBoard.Note that you may run out of memory if you run the this script on the same GPU
as the training script. You can set the environment variableCUDA_VISIBLE_DEVICES=""to force the script to run on CPU. If it runs too
slowly on CPU, you can decrease the value of--num_eval_examples.DATA_DIR="${HOME}/skip_thoughts/data"MODEL_DIR="${HOME}/skip_thoughts/model"# Ignore GPU devices (only necessary if your GPU is currently memory# constrained, for example, by running the training script).exportCUDA_VISIBLE_DEVICES=""# Run the evaluation script. This will run in a loop, periodically loading the# latest model checkpoint file and computing evaluation metrics.bazel-bin/skip_thoughts/track_perplexity\--input_file_pattern="${DATA_DIR}/validation-?????-of-00001"\--checkpoint_dir="${MODEL_DIR}/train"\--eval_dir="${MODEL_DIR}/val"\--num_eval_examples=50000If you started thetrack_perplexityscript, run aTensorBoardserver in a separate process for real-time monitoring of training summaries and
validation perplexity.MODEL_DIR="${HOME}/skip_thoughts/model"# Run a TensorBoard server.tensorboard--logdir="${MODEL_DIR}"Expanding the VocabularyOverviewThe vocabulary generated by the preprocessing script contains only 20,000 words
which is insufficient for many tasks. For example, a sentence from Wikipedia
might contain nouns that do not appear in this vocabulary.A solution to this problem described in theSkip-Thought Vectorspaper is to learn a mapping that transfers word representations from one model to
another. This idea is based on the "Translation Matrix" method from the paperExploiting Similarities Among Languages for Machine Translation.Specifically, we will load the word embeddings from a trainedSkip-Thoughtsmodel and from a trainedword2vec model(which has a much larger vocabulary). We will train a linear regression model
without regularization to learn a linear mapping from the word2vec embedding
space to theSkip-Thoughtsembedding space. We will then apply the linear
model to all words in the word2vec vocabulary, yielding vectors in theSkip-
Thoughtsword embedding space for the union of the two vocabularies.The linear regression task is to learn a parameter matrixWto minimize|| X - Y * W ||2, whereXis a matrix ofSkip-Thoughtsembeddings of shape[num_words, dim1],Yis a matrix of word2vec embeddings
of shape[num_words, dim2], andWis a matrix of shape[dim2, dim1].PreparationFirst you will need to download and unpack a pretrainedword2vec modelfromthis website(direct download link).
This model was trained on the Google News dataset (about 100 billion words).Also ensure that you have alreadyinstalled gensim.Run the Vocabulary Expansion Script# Path to checkpoint file or a directory containing checkpoint files (the script# will select the most recent).CHECKPOINT_PATH="${HOME}/skip_thoughts/model/train"# Vocabulary file generated by the preprocessing script.SKIP_THOUGHTS_VOCAB="${HOME}/skip_thoughts/data/vocab.txt"# Path to downloaded word2vec model.WORD2VEC_MODEL="${HOME}/skip_thoughts/googlenews/GoogleNews-vectors-negative300.bin"# Output directory.EXP_VOCAB_DIR="${HOME}/skip_thoughts/exp_vocab"# Build the vocabulary expansion script.cdtensorflow-models/skip_thoughts
bazelbuild-copt//skip_thoughts:vocabulary_expansion# Run the vocabulary expansion script.bazel-bin/skip_thoughts/vocabulary_expansion\--skip_thoughts_model=${CHECKPOINT_PATH}\--skip_thoughts_vocab=${SKIP_THOUGHTS_VOCAB}\--word2vec_model=${WORD2VEC_MODEL}\--output_dir=${EXP_VOCAB_DIR}Evaluating a ModelOverviewThe model can be evaluated using the benchmark tasks described in theSkip-Thought Vectorspaper. The following tasks are supported (refer to the paper for full details):SICKsemantic relatedness task.MSRP(Microsoft Research Paraphrase Corpus) paraphrase detection task.Binary classification tasks:MRmovie review sentiment task.CRcustomer product review task.SUBJsubjectivity/objectivity task.MPQAopinion polarity task.TRECquestion-type classification task.PreparationYou will need to clone or download theskip-thoughts GitHub repositorybyryankiros(the first author of the Skip-Thoughts
paper):# Folder to clone the repository to.ST_KIROS_DIR="${HOME}/skip_thoughts/skipthoughts_kiros"# Clone the repository.gitclonegit@github.com:ryankiros/skip-thoughts.git"${ST_KIROS_DIR}/skipthoughts"# Make the package importable.exportPYTHONPATH="${ST_KIROS_DIR}/:${PYTHONPATH}"You will also need to download the data needed for each evaluation task. See the
instructionshere.For example, the CR (customer review) dataset is foundhere. For this task we want the
filescustrev.posandcustrev.neg.Run the Evaluation TasksIn the following example we will evaluate a unidirectional model ("uni-skip" in
the paper) on the CR task. To use a bidirectional model ("bi-skip" in the
paper),  simply pass the flags--bi_vocab_file,--bi_embeddings_fileand--bi_checkpoint_pathinstead. To use the "combine-skip" model described in the
paper you will need to pass both the unidirectional and bidirectional flags.# Path to checkpoint file or a directory containing checkpoint files (the script# will select the most recent).CHECKPOINT_PATH="${HOME}/skip_thoughts/model/train"# Vocabulary file generated by the vocabulary expansion script.VOCAB_FILE="${HOME}/skip_thoughts/exp_vocab/vocab.txt"# Embeddings file generated by the vocabulary expansion script.EMBEDDINGS_FILE="${HOME}/skip_thoughts/exp_vocab/embeddings.npy"# Directory containing files custrev.pos and custrev.neg.EVAL_DATA_DIR="${HOME}/skip_thoughts/eval_data"# Build the evaluation script.cdtensorflow-models/skip_thoughts
bazelbuild-copt//skip_thoughts:evaluate# Run the evaluation script.bazel-bin/skip_thoughts/evaluate\--eval_task=CR\--data_dir=${EVAL_DATA_DIR}\--uni_vocab_file=${VOCAB_FILE}\--uni_embeddings_file=${EMBEDDINGS_FILE}\--uni_checkpoint_path=${CHECKPOINT_PATH}Output:[0.82539682539682535,0.84084880636604775,0.83023872679045096,0.86206896551724133,0.83554376657824936,0.85676392572944293,0.84084880636604775,0.83023872679045096,0.85145888594164454,0.82758620689655171]The output is a list of accuracies of 10 cross-validation classification models.
To get a single number, simply take the average:ipython# Launch iPython.In[0]:importnumpyasnpnp.mean([0.82539682539682535,0.84084880636604775,0.83023872679045096,0.86206896551724133,0.83554376657824936,0.85676392572944293,0.84084880636604775,0.83023872679045096,0.85145888594164454,0.82758620689655171])Out[0]:0.84009936423729525Encoding SentencesIn this example we will encode data from themovie review dataset(specifically thesentence polarity dataset v1.0).ipython# Launch iPython.In[0]:# Imports.from__future__importabsolute_importfrom__future__importdivisionfrom__future__importprint_functionimportnumpyasnpimportos.pathimportscipy.spatial.distanceassdfromskip_thoughtsimportconfigurationfromskip_thoughtsimportencoder_managerIn[1]:# Set paths to the model.VOCAB_FILE="/path/to/vocab.txt"EMBEDDING_MATRIX_FILE="/path/to/embeddings.npy"CHECKPOINT_PATH="/path/to/model.ckpt-9999"# The following directory should contain files rt-polarity.neg and# rt-polarity.pos.MR_DATA_DIR="/dir/containing/mr/data"In[2]:# Set up the encoder. Here we are using a single unidirectional model.# To use a bidirectional model as well, call load_model() again with# configuration.model_config(bidirectional_encoder=True) and paths to the# bidirectional model's files. The encoder will use the concatenation of# all loaded models.encoder=encoder_manager.EncoderManager()encoder.load_model(configuration.model_config(),vocabulary_file=VOCAB_FILE,embedding_matrix_file=EMBEDDING_MATRIX_FILE,checkpoint_path=CHECKPOINT_PATH)In[3]:# Load the movie review dataset.data=[]withopen(os.path.join(MR_DATA_DIR,'rt-polarity.neg'),'rb')asf:data.extend([line.decode('latin-1').strip()forlineinf])withopen(os.path.join(MR_DATA_DIR,'rt-polarity.pos'),'rb')asf:data.extend([line.decode('latin-1').strip()forlineinf])In[4]:# Generate Skip-Thought Vectors for each sentence in the dataset.encodings=encoder.encode(data)In[5]:# Define a helper function to generate nearest neighbors.defget_nn(ind,num=10):encoding=encodings[ind]scores=sd.cdist([encoding],encodings,"cosine")[0]sorted_ids=np.argsort(scores)print("Sentence:")print("",data[ind])print("\nNearest neighbors:")foriinrange(1,num+1):print("%d.%s(%.3f)"%(i,data[sorted_ids[i]],scores[sorted_ids[i]]))In[6]:# Compute nearest neighbors of the first sentence in the dataset.get_nn(0)Output:Sentence:
 simplistic , silly and tedious .

Nearest neighbors:
 1. trite , banal , cliched , mostly inoffensive . (0.247)
 2. banal and predictable . (0.253)
 3. witless , pointless , tasteless and idiotic . (0.272)
 4. loud , silly , stupid and pointless . (0.295)
 5. grating and tedious . (0.299)
 6. idiotic and ugly . (0.330)
 7. black-and-white and unrealistic . (0.335)
 8. hopelessly inane , humorless and under-inspired . (0.335)
 9. shallow , noisy and pretentious . (0.340)
 10. . . . unlikable , uninteresting , unfunny , and completely , utterly inept . (0.346)