Question: What baselines are used in experiments?
Evidence: Optimizations ::: a) Parallel Scan Inference
The commutative properties of semiring algorithms allow flexibility in the order in which we compute $A(\ell )$. Typical implementations of dynamic programming algorithms are serial in the length of the sequence. On parallel hardware, an appealing approach is a parallel scan ordering BIBREF35, typically used for computing prefix sums. To compute, $A(\ell )$ in this manner we first pad the sequence length $T$ out to the nearest power of two, and then compute a balanced parallel tree over the parts, shown in Figure FIGREF21. Concretely each node layer would compute a semiring matrix multiplication, e.g. $ \bigoplus _c \ell _{t, \cdot , c} \otimes \ell _{t^{\prime }, c, \cdot }$. Under this approach, we only need $O(\log N)$ steps in Python and can use parallel GPU operations for the rest. Similar parallel approach can also be used for computing sequence alignment and semi-Markov models.
Optimizations ::: b) Vectorized Parsing
Computational complexity is even more of an issue for parsing algorithms, which cannot be as easily parallelized. The log-partition for parsing is computed with the Inside algorithm. This algorithm must compute each width from 1 through T in serial; however it is important to parallelize each inner step. Assuming we have computed all inside spans of width less than $d$, computing the inside span of width $d$ requires computing for all $i$,
Optimizations ::: c) Semiring Matrix Operations
The two previous optimizations reduce most of the cost to semiring matrix multiplication. In the specific case of the $(\sum , \times )$ semiring these can be computed very efficiently using matrix multiplication, which is highly-tuned on GPU hardware. Unfortunately for other semirings, such as log and max, these operations are either slow or very memory inefficient. For instance, for matrices $T$ and $U$ of sized $N \times M$ and $M \times O$, we can broadcast with $\otimes $ to a tensor of size $N \times M \times O$ and then reduce dim $M$ by $\bigoplus $ at a huge memory cost. To avoid this issue, we implement custom CUDA kernels targeting fast and memory efficient tensor operations. For log, this corresponds to computing,
Parallel Scan Inference
The commutative properties of semiring algorithms allow flexibility in the order in which we compute $A(\ell )$. Typical implementations of dynamic programming algorithms are serial in the length of the sequence.
Vectorized Parsing
Computational complexity is even more of an issue for parsing algorithms, which cannot be as easily parallelized.
Semiring Matrix Operations
The two previous optimizations reduce most of the cost to semiring matrix multiplication. In the specific case of the $(\sum , \times )$ semiring these can be computed very efficiently using matrix multiplication, which is highly-tuned on GPU hardware. Unfortunately for other semirings, such as log and max, these operations are either slow or very memory inefficient. For instance, for matrices $T$ and $U$ of sized $N \times M$ and $M \times O$, we can broadcast with $\otimes $ to a tensor of size $N \times M \times O$ and then reduce dim $M$ by $\bigoplus $ at a huge memory cost.
Answer: The answers are shown as follows:
* Typical implementations of dynamic programming algorithms are serial in the length of the sequence
* Computational complexity is even more of an issue for parsing algorithms, which cannot be as easily parallelized
* Unfortunately for other semirings, such as log and max, these operations are either slow or very memory inefficient