Question: What details are key for reproducing results related to DecAtt?
Evidence: We implement all the models with the same PyTorch framework. Below, we summarize the implementation details that are key for reproducing results for each model:
SSE: This model can converge very fast, for example, 2 or 3 epochs for the SNLI dataset. We control the convergence speed by updating the learning rate for each epoch: specifically, $lr=\frac{1}{2^{\frac{epoch\_i}{2}}}*{init\_lr}$ , where $init\_lr$ is the initial learning rate and $epoch\_i$ is the index of current epoch.
DecAtt: It is important to use gradient clipping for this model: for each gradient update, we check the L2 norm of all the gradient values, if it is greater than a threshold $b$ , we scale the gradient by a factor $\alpha = b/L2\_norm$ . Another useful procedure is to assemble batches of sentences with similar length.
ESIM: Similar but different from DecAtt, ESIM batches sentences with varied length and uses masks to filter out padding information. In order to batch the parse trees within Tree-LSTM recursion, we follow Bowman et al.'s bowman-EtAl:2016:P16-1 procedure that converts tree structures into the linear sequential structure of a shift reduce parser. Two additional masks are used for producing left and right children of a tree node.
PWIM: The cosine and Euclidean distances used in the word interaction layer have smaller values for similar vectors while dot products have larger values. The performance increases if we add a negative sign to make all the vector similarity measurements behave consistently.
We implement all the models with the same PyTorch framework. Below, we summarize the implementation details that are key for reproducing results for each model:
SSE: This model can converge very fast, for example, 2 or 3 epochs for the SNLI dataset. We control the convergence speed by updating the learning rate for each epoch: specifically, $lr=\frac{1}{2^{\frac{epoch\_i}{2}}}*{init\_lr}$ , where $init\_lr$ is the initial learning rate and $epoch\_i$ is the index of current epoch.
DecAtt: It is important to use gradient clipping for this model: for each gradient update, we check the L2 norm of all the gradient values, if it is greater than a threshold $b$ , we scale the gradient by a factor $\alpha = b/L2\_norm$ . Another useful procedure is to assemble batches of sentences with similar length.
ESIM: Similar but different from DecAtt, ESIM batches sentences with varied length and uses masks to filter out padding information. In order to batch the parse trees within Tree-LSTM recursion, we follow Bowman et al.'s bowman-EtAl:2016:P16-1 procedure that converts tree structures into the linear sequential structure of a shift reduce parser. Two additional masks are used for producing left and right children of a tree node.
PWIM: The cosine and Euclidean distances used in the word interaction layer have smaller values for similar vectors while dot products have larger values. The performance increases if we add a negative sign to make all the vector similarity measurements behave consistently.
We implement all the models with the same PyTorch framework.23 Below, we summarize the implementation details that are key for reproducing results for each model: ****•DecAtt: It is important to use gradient clipping for this model: for each gradient update, we check the L2 norm of all the gradient values, if it is greater than a threshold b, we scale the gradient by a factor a = b/L2-norm. Another useful procedure is to assemble batches of sentences with similar length.
Answer: It is important to use gradient clipping for this model: for each gradient update, they check the L2 norm of all the gradient values, if it is greater than a threshold b, they scale the gradient by a factor a = b/L2-norm. Another useful procedure is to assemble batches of sentences with similar length.