(2024)

Abstract: Recently, binary representation has been proposed as a novel representation that lies between continuous and discrete representations. It exhibits considerable information-preserving capability when being used to replace continuous input vectors. In this paper, we investigate the feasibility of further introducing it to the output side, aiming to allow models to output binary labels instead. To preserve the structural information on the output side along with label information, we extend the previous contrastive hashing method as structured contrastive hashing. More specifically, we upgrade CKY from label-level to bit-level, define a new similarity function with span marginal probabilities, and introduce a novel contrastive loss function with a carefully designed instance selection strategy. Our model1 achieves competitive performance on various structured prediction tasks, and demonstrates that binary representation can be considered a novel representation that further bridges the gap between the continuous nature of deep learning and the discrete intrinsic property of natural languages.

heading: 1 Introduction
text: Bridging the gap between the continuous nature of deep learning and the discrete intrinsic property of natural languages has been one of the most fundamental and essential questions since the very beginning. Continuous representation makes the training of neural networks effective and efficient. Nowadays, representing discrete natural languages in continuous format is the first and foremost step to leveraging the capabilities of deep learning. One could even argue that the exhilarating advancements in natural language processing in the past decade can largely be attributed to the word embedding technique, as it is the first successful attempt.
Word embedding (Bengio et al., 2000; Mikolov et al., 2013a,b) technique replaces the vocabulary-
1https://github.com/speedcell4/parserker
sized one-hot word representations with compact continuous vectors. Since then, input and output layers incorporating embedding matrices have be-
8036
come standard components in neural models. Discrete tokens are mapped into continuous vectors by looking up the corresponding index in it, and continuous vectors are mapped back to discrete tokens by searching the most similar one from it. However, the essence of this operation is still one-hot encoding, even the following subword tokenization techniques (Sennrich et al., 2016; Kudo, 2018) attempt to mitigate this issue by decomposing words into subword units, these approaches still require the building vocabularies and embedding matrices that consists of tens of thousands of tokens. In the era of large language models (OpenAI et al., 2023; Touvron et al., 2023), these embedding matrices typically account for a considerable number of parameters, especially in cross-lingual models. Besides, parameter updates also solely rely on the sparse gradients backpropagated to the limited tokens present in sentences. Moreover, imposing structural constraints on continuous representations to model relations among tokens is considered difficult, whereas it is easy and common in discrete representations. Therefore, further bridging the gap has become increasingly important nowadays.
Recently, Wang et al. (2023) introduced a novel binary representation that lies between continuous and discrete representations. They proposed a contrastive hashing method to compress continuous hidden states into binary codes. These codes contain all the necessary task-relevant information, and using them as the only inputs can reproduce the performance of the original models. Unlike associating each token with only a single vector, their method allocates multiple bits to each token, and the token representation can be constructed by concatenating these bit vectors. In other words, their binary representation breaks tokens down into combinations of semantic subspaces. As a result, replacing the token embedding matrix in the input layer with a tiny bit embedding matrix without sacrificing performance becomes possible.
In this paper, we explore the possibility of further introducing this representation to output layers. In the input layer, structural information can only be implicitly obtained by introducing the task loss as an auxiliary. However, the output layers often involve complex intra-label constraints, especially for structured prediction tasks, structural information can and should be explicitly preserved along with plain label information. Therefore, we attempt to endow models with this capability by extending previous contrastive hashing to structured
contrastive hashing. We begin by upgrading the CKY, which parses sentences, returns spans with discrete labels, to support binary format labels (§3.1). Subsequently, we define a new similarity function by using span marginal probabilities obtained from this bit-level CKY (§3.2) to jointly learn label and structural information. Furthermore, we conduct a detailed analysis of several widely used contrastive learning losses, identifying the geometric center issue, and introduce a novel contrastive learning loss to remedy it (§3.3) through carefully selecting instances. By doing so, we show that it is feasible to introduce binary representation to output layers and have them output binary labels on trees. Moreover, since our model is based on contrastive learning, it also benefits from its remarkable representation learning capability, resulting in better performance than existing models. We conduct experiments on constituency parsing and nested named entity recognition. Experimental results (§4.2) demonstrate that our models achieve competitive performance with only around 12 and 8 bits, respectively.

heading: 2 Background

heading: 2.1 Constituency Parsing
text: For a given sentence w1, . . . , wn, constituency parsing aims at detecting its hierarchical syntactic structures. Previous work (Stern et al., 2017; Gaddy et al., 2018; Kitaev and Klein, 2018; Kitaev et al., 2019; Zhang et al., 2020) decompose tree score g(t) as the sum of its constituents scores,
g(t) = ∑
⟨li,ri,yi⟩∈t g(li, ri, yi) (1)
where li and ri indicate the left and right boundary of the i-th span, and yi ∈ Y stands for the label. Constituent score g(li, ri, yi) reflects the joint score
of selecting the specified span and assigning it the specified label. Previous work (Kitaev and Klein, 2018; Zhang et al., 2020) commonly compute this score using a linear or bilinear component. Under the framework of graphical probabilistic models, they can efficiently compute the conditional probability by applying the CKY algorithm.
p(t) = exp g(t)
Z ≡∑t′∈T exp g(t′) (2)
Z is commonly known as the partition function which enumerates all valid constituency trees.
Besides, marginal probability µ(li, ri, yi) is also frequently mentioned. It stands for the proportion of scores for all trees that include the specified span with the specified label. As noted by Eisner (2016), computing the partial derivative of the log partition with respect to the span score is an efficient approach to obtain the marginal probability.
µ(li, ri, yi) = ∂ logZ
∂ g(li, ri, yi) (3)
Intuitively, marginal probability indicates the joint probability of selecting a specified span with a specified label. Therefore, it is easy to notice that merely summing the marginal probabilities for all labels of a given span does not always yield 1, i.e., ∑ y′∈Y µ(li, ri, y
′) ̸≡ 1, as there is no guarantee that this span will be selected. In other words, marginal probabilities contain not only label information but also structural information. If a span is unlikely to be selected, its marginal probability will not be high regardless of the label.

heading: 2.2 Contrastive Hashing
text: Contrastive learning (He et al., 2020; Gao et al., 2021; Khosla et al., 2020) is an effective yet simple representation learning method, which involves pulling together positive pairs and pushing apart negative pairs in a metric space. Recently, Wang et al. (2023) extended this approach as contrastive hashing. They append an untrained transformer to the end of a pre-trained language model and use its attention scores for both task learning and hashing. Specifically, its entire attention probabilities aki,j are used to compute hidden states for downstream tasks as usual, and its diagonal entries ski,i of the attention scores are employed for hashing.
ski,j = (WQk hi) ⊤(WKk hj)√ dk
(4)
aki,j = softmax j (ski,j) (5)
Where WQk and W K k are parameters, h are hidden states, dk is the head dimension. These two learning objectives share the same attention matrix, therefore, task-relevant information is implicitly ensured to be preserved in these binary codes.
More specifically, to leverage the multi-head mechanism, they allow each head to represent one and only one bit. By increasing the number of heads to K, they obtain attention scores from K different semantic aspects. During the inference stage, codes are generated by binarizing these scores,
ci = [c 1 i , · · · , cKi ] ∈ {−1,+1}K (6) cki = sign (s k i,i)
During the training stage, they approximate Hamming similarity by computing the cosine similarity, with one of its inputs binarized first.
s(i, j) = cos (si,i, cj) (7)
Apart from this similarity function, they also propose a novel loss by carefully selecting instances and eliminating potential positives and negatives. They fine-tune the entire model using both the downstream task loss and the contrastive hashing loss, i.e., L = Ltask + β · Lcontrastive. Experiments show that they can reproduce the original performance on an extremely tiny model using only these 24-bit codes as inputs. Therefore, they claim that these codes preserve all the necessary task-relevant information.

heading: 3 Proposed Methods
text: Our model attempts to learn parsing and hashing simultaneously with a single structured contrastive hashing loss. In other words, we try to introduce the binary representation to output layers and eliminate the need for the Ltask above. To achieve this, we first extend the CKY module to support binary labels (§3.1). Then, we replace the cosine similarity with a newly defined similarity function (§3.2) based on span marginal probabilities, because it contains not only label information but also structural information. After that, we analyze several commonly used contrastive losses, and propose a new one (§3.3) to mitigate the geometric center issue as shown in Figure 2. After training, we build code vocabulary by mapping binary codes back to their most frequently coinciding labels.

heading: 3.1 Constituency Parsing with Bits
text: We decompose tree scores as the sum of constituent scores as well, but with discrete labels yi replaced with binary codes ci ∈ {−1,+1}K .
g(t) = ∑
⟨li,ri,ci⟩∈t g(li, ri, ci) (8)
g(li, ri, ci) = K∑
k=1
gk(li, ri, c k i ) (9)
Where gk(li, ri, cki ) represents the span score with the k-th bit position assigned as value cki . We additionally assume that the bits are independent of each other, so we simply add their scores together to obtain the span score g(li, ri, ci).
Following Wang et al. (2023), we maintain the one-head-one-bit design and also utilize attention scores for hashing. Furthermore, since we attempt to eliminate Ltask, we do not need to compute the final outputs of the transformer layer. Therefore, we only retain the query WQk and key W K k to calculate the span score for getting +1 in the k-th bit position by using the token hidden states of the left and right span boundary hli and hri . For the −1 case, we simply leave its score as 0.
gk(li, ri,+1) = (WQk hli) ⊤(WKk hri)√ dk
(10)
gk(li, ri,−1) = 0 (11) With these definitions, we can extend the CKY module to the bit-level and calculate the conditional probability and partition function using Equation 2 as usual. Additionally, the bit-level marginal probability is defined as below.
µk(li, ri, c k i ) =
∂ logZ
∂ gk(li, ri, c k i )
(12)

heading: 3.2 Contrastive Hashing with Structures
text: Wang et al. (2023) emphasize that the key to their loss function is first hashing one of its inputs as codes and then calculating the similarity between continuous scores and the discrete codes. We define our similarity function in a similar way. Since we can straightforwardly obtain the span marginal probabilities with Equation 12, we then binarize scores into codes towards the sides with the higher span marginal probabilities.
ci = [c 1 i , . . . , c K i ] ∈ {−1,+1}K (13)
cki = { +1 µk(li, ri,+1) > µk(li, ri,−1) −1 otherwise
Naturally, we define the similarity function between two spans as the marginal probability of selecting i-th span while assigning cj as its code.
s(i, j) = 1
K
K∑
k=1
µk(li, ri, c k j ) (14)
As we mentioned above (§2.1), we use marginal probabilities to define the similarity function because they reflect the joint probability of both selecting the specified span and assigning the specified label to it. If a span is unlikely to be selected as a phrase, then both µk(li, ri,+1) and µk(li, ri,−1) will be close to zero. Thus, the model learns structural and label information simultaneously. By leveraging this similarity function, we extend contrastive hashing to structured contrastive hashing. This approach eliminates the label embedding from the output layer, as the hashing layer returns labels in binary format now.
Moreover, for a sentence with n tokens, the total number of spans is (n2 + n)/2, and bit-level CKY returns marginal probabilities for them all. However, using them for contrastive hashing leads to an intractable time complexity of O(n4). In practice, we select only spans from the target trees, reducing the number of spans to 2n− 1 to maintain the time complexity at O(n2). This is another reason why we prefer marginal probability.

heading: 3.3 Instance Selection
text: Following the contrastive learning framework (Gao et al., 2021; Wang et al., 2023), we feed sentences into the neural network twice to obtain two semantically identical but slightly augmented representations. In this way, we get two different marginal probabilities for each span. We calculate contrastive losses by comparing each span across views and average them as the batch loss (§3.5). Note that each batch contains spans from multiple sentences, so our contrastive hashing also compares spans across different sentences. For clarity, we omit subscripts in the following equations when there is no ambiguity.
As we mentioned above (§2.2), the fundamental concept of contrastive learning is pulling together positives and pushing apart negatives. The most commonly used objective function is defined as,
Lself = − log exp s(i, i)∑
j∈N∪P exp s(i, j) (15)
N = {j | yj ̸= yi} and P = {j | yj = yi} stands for the negative and positive sets, respectively. We
additionally define S = {i} as the set that contains span i as its only entry. It is obvious that S ⊆ P always holds.
In addition, the log ∑
exp operator is commonly considered a differentiable approximation of the max operator. Therefore, by slightly tweaking the equation, we reinterpret it as below.
Lself = log ∑
j
exp s(i, j)− s(i, i)
≈ max j∈N∪P s(i, j) − s(i, i) (16)
Moreover, Khosla et al. (2020) also proposed a loss function for supervised settings where multiple positives are present. By applying the same tricks, we can rewrite the loss function as follows.
Lsup = − 1 |P| ∑
p∈P log exp s(i, p)∑ j∈N∪P exp s(i, j)
= log ∑
j
exp s(i, j)− 1|P| ∑
p
s(i, p)
≈ max j∈N∪P s(i, j) − mean j∈P s(i, j) (17)
Wang et al. (2023) have also proposed a contrastive loss function for hashing. They claim that identical tokens may even not contain identical information due to different contexts. Therefore, they treat P as potential false positives and negatives, and replace it with S in both terms.
Lhash = − log exp s(i, i)∑
j∈N∪S exp s(i, j)
≈ max j∈N∪S s(i, j) − s(i, i) (18)
By unifying them all in a common format, we can observe that their main differences lie in the instance selection strategies. Both Lself and Lsup pull instances towards the geometric center of their positive instances in the second term . The only difference is that Lself assumes there is only one positive instance, making the geometric center merely i-th span itself, whereas Lsup has access to the groundtruth labels and thus can obtain a more specific center. On the other hand, Lhash differs from Lself in the first term , as it suggests that dividing instances solely based on ground-truth labels may also introduce false positives and negatives. Therefore, Lhash excludes potential false positives from this term. Additionally, it is noteworthy that among these three losses, all first terms employ max, while
all second terms use mean. Intuitively speaking, the max operator pulls towards the most likely true positive instance, while mean operator pulls towards the geometric center of all positive instances. Although it is hard to determine whether instances in P are false positives or not, what we can be certain of is that there is at least one true positive instance, since S ⊆ P always holds. Therefore, using the max operator to pull towards the most probable one is a more effective approach.
Lmax ≈ max j∈N∪S s(i, j) − max j∈P s(i, j)
≈ log ∑
j∈N∪S exp s(i, j)
− log ∑
p∈P exp s(i, p)
= − log ∑
p∈P exp s(i, p)∑ j∈N∪S exp s(i, j)
(19)

heading: 3.4 Architecture
text: As shown in Figure 1, our model consists of a pretrained language model, an attention hash layer, and a bit-level CKY. The attention hash layer derives from the transformer layer, but it preserves only the query and key components for calculating span score g(li, ri, ci). All the other components, such as layer normalization (Ba et al., 2016) and feed-forward layers, are removed, as there is no need to calculate hidden states.
Compared with existing constituency parsers (Kitaev and Klein, 2018; Zhang et al., 2020), the output layer of our parser does not include label embedding matrices. Instead, it utilizes the attention hash layer and a bit-level CKY to predict the binary representation of labels. Consequently, the number of parameters of this part changes from |Y| × d to two K × ⌈ d K ⌉ × d.

heading: 3.5 Training and Inference
text: During the training stage, we feed sentences into the model twice to obtain marginal probabilities µ1 and µ2 of the two views, and binarize spans on the target trees t̂1 = {⟨li, ri, c1i ⟩}2n−1i=1 and t̂2 = {⟨li, ri, c2i ⟩}2n−1i=1 using Equation 13, respectively. Besides, we use the ground-truth labels yi to divide N and P . After that, we calculate the contrastive hashing loss for each span with Equation 19 and average them as the batch loss.
L = mean 1≤i<2n
( L(i, µ1, t̂2) + L(i, µ2, t̂1) ) (20)
and the second-best performance of each column. ♭♯♢ stands for the graph-based, transition-based, and sequence-to-sequence models, respectively.
After training, we switch the model to evaluation mode, i.e., turning off dropouts (Srivastava et al., 2014), and then feed all the training sentences into the model again. During this pass, we count the frequency of each pair of binary code and its corresponding ground-truth label, i.e., f(c, y). Then, we can reconstruct a code vocabulary to map codes back to their most frequently coinciding labels.
yc ← argmax y∈Y f(c, y) (21)
During the inference stage, we search the most probable tree from all valid trees using the CockeKasami-Younger (CKY) algorithm (Kasami, 1966). Our bit-level parsers return span boundaries and their binary codes, we translate them back to labels by using the Equation above.

heading: 4 Experiments

heading: 4.1 Settings
text: We validate the effectiveness of our model on various structured prediction tasks, i.e., constituency parsing and nested named entity recognition tasks. Dataset statistics can be found in Appendix A.
For the constituency parsing task, we conduct experiments on the datasets PTB (Marcus et al.,
1993) and CTB5.1 (Xue et al., 2005). We transform the original trees into those of Chomsky normal form and adopt left binarization with NLTK (Bird and Loper, 2004). We study model performance by employing pre-trained language models with checkpoints bert-large-cased (Devlin et al., 2019) and xlnet-large-cased (Yang et al., 2019) for PTB, and bert-base-chinese for CTB.
For nested named entity recognition task, we use datasets ACE2004 (Doddington et al., 2004), ACE2005 (Walker et al., 2006), and GENIA (Kim et al., 2003). We follow the data splitting of Shibuya and Hovy (2020). Nested named entity recognition, as Fu et al. (2021) claimed, can be considered as a partially observed constituency parsing task. Therefore, we add a dummy span TOP as the top span to each sentence to ensure all the observed spans form a valid parsing tree, and apply the same transformation and binarization on it. We use the checkpoint bert-large-cased on ACE2004 and ACE2005, and for the GENIA dataset we use dmis-lab/biobert-large-cased-v1.1 (Lee et al., 2019) as the pre-trained language model.
We utilize the deep learning framework PyTorch (Paszke et al., 2019) to implement our models and download pre-trained lanugage checkpoints from huggingface/transformers (Wolf et al., 2020).
To keep the training of contrastive hashing stable, we collect sentences until the total number of tokens in each batch reaches 1024. We employ Adam
optimizer (Kingma and Ba, 2015; Loshchilov and Hutter, 2019), and the total number of training steps of constituency parsing and nested named entity recognition are 50,000 and 20,000, and the warm-up are 4000 and 2000 steps, respectively. To provide harder negatives by augmenting inputs, we also randomly mask a portion of tokens as [MASK].
Experiments are all conducted on a single NVIDIA Tesla V100 graphics card, the total training wall time is around 3 hours and 1 hour, respectively. All experiments are run with two different random seeds and the reported numbers in the following tables are their averages.

heading: 4.2 Main Results
text: Our model consistently achieves competitive performance on various structured prediction tasks and datasets, as presented in Table 1 and Table 2.
For constituency parsing, our models reach the peak with around 12 bits. Continuously increasing the number of bits does not further improve performance, on the contrary, it leads to a slight decline. We attribute this to the disproportionately large hashing space, as the amount of information carried by each task and dataset is fixed. For example, assigning K bits to a task with only K labels leads to an extreme case. Models in this case tend to produce the most trivial bit-level one-hot representation, making them nothing different from traditional static embedding models. On the contrary, decreasing the number of bits to fewer than 8 bits is also harmful, due to the insufficient representation capability. Besides, our models outperform almost all previous graph-based methods that rely on maximizing the log-likelihoods of target trees (Kitaev et al., 2019; Mrini et al., 2020; Zhang et al., 2020; Xin et al., 2021; Cui et al., 2022). There-
fore, we claim that leveraging contrastive learning is beneficial to representation learning.
For nested named entity recognition, all datasets show the best performance at the 8-bit settings, and decreasing to fewer than 6 bits also results in insufficient representing capability. Similarly, our methods outperform the previous sequential labeling methods (Shibuya and Hovy, 2020; Wang et al., 2021; Xin et al., 2021) and graph-based methods (Fu et al., 2021; Lou et al., 2022). In addition, some other papers (Yu et al., 2020; Shen et al., 2021; Tan et al., 2021; Zhu and Li, 2022) propose to straightforwardly enumerate all spans and directly train the model to classify them. These methods currently show the best performance, our method can also achieve comparable results to them.

heading: 4.3 Ablation Studies
text: We enumerate all combinations of the contrastive learning objective functions in Table 3 to study the influence of instance selection strategies.
We note that selecting N∪S is always superior to
S[D88]
VP[8A7]
NP[54D]
selecting N∪P in the negative term , but using S is inferior to using P in the positive term . This leads to the conclusion that adding positive instances P to the negative term is harmful, but it turns out to be beneficial when added to the positive term. This seems contradictory to the conclusion of Wang et al. (2023). However, this discrepancy arises because the training mechanisms are different. Their model relies on downstream task loss to implicitly learn structural information, whereas our model is designed to learn it explicitly. Therefore, we claim for the setting that aims at straightforwardly hashing both the label and structural information using a single loss, our Lmax is the best choice.
Besides, Ouchi et al. (2020, 2021) appear to be the most relevant publications to our research. They also attempt to directly apply contrastive learning methods to structured prediction tasks. However, they pull instances towards the geometric centers, and their results show that this does not lead to noteworthy improvements than conventional graphbased models. This conclusion is consistent with ours, i.e., maxP is better than meanP . We hypothesize that calculating the geometric center largely erases the contextual information of each instance. Pulling towards this center is not essentially different from pulling towards a static label embedding. Therefore, their approach loses the advantage of contrastive learning and trivially falls back to the conventional static embedding classification. On the contrary, our maxP primarily targets the closest instance instead, therefore, it can fully leverage the capability of contrastive learning and distribute instances more uniformly, and allows each label to associate with multiple codes.

heading: 4.4 Case Studies
text: We show some frequent labels and their associated codes in Table 4. We only display the codes that
occupy the top 90% frequency of each label. It can be observed that some labels have two codes with roughly the same frequency, while others are predominantly represented by a single code. However, even in these cases, none reach 100% coverage. This result further demonstrates that the term maxP imposes only a weak supervision signal and does not force all instances towards the geometric centers, thereby avoiding false positives and allowing a more uniform distribution of instances, rather than crowding at a few specific points.
In Figure 3, we display some parsing examples, using the same sentences as Wang et al. (2023). Our parser takes discrete tokens as inputs and returns binary labels, while theirs, conversely, receives binary codes of tokens and parses them into trees with discrete labels. Our codes preserve not only static by also contextual information of each label, for instance, the two S labels above are translated from two different codes. Therefore, our methods can also be considered as implicitly clustering methods, since these two codes refer to two different sub-labels of label S. This phenomenon is unlikely to be observed in models suffering from the geometric center issue.

heading: 5 Related Work
text: The ease of training has allowed contiguous representation (Hinton and Salakhutdinov, 2006) to achieve great success in applications such as masked language models (Devlin et al., 2019; Liu et al., 2020) and causal language models (Peters et al., 2018; Lewis et al., 2020; OpenAI et al., 2023; Touvron et al., 2023). Different from them, autoencoder (Vincent et al., 2010; Kingma and Welling, 2014; Rezende et al., 2014) and contrastive learning (He et al., 2020; Gao et al., 2021) do not classify instances into specific classes, but rather reconstruct the input or determine whether a given pair
belongs to the same class. The advantage of these approaches is that they do not require introducing embedding matrices, making representation learning more effective. However, because embeddings are not introduced, continuous vectors are hard to be discretized to specific classes and need to train an additional classifier for such purpose.
Learning discrete representation has also been widely focused on since the early era of deep learning (Salakhutdinov and Hinton, 2009; Courville et al., 2011; Mnih and Gregor, 2014; Mnih and Rezende, 2016), as it is a necessary component of many downstream applications. For example, languages are inherently discrete, and mapping discrete tokens to continuous vectors and back is the foremost step for modern natural language processing. Additionally, many applications involving the control and interpretation of intermediate variables also require interleaving non-differentiable layers. However, the key challenge is that backpropagating gradients through discrete representation is generally intractable, making them hard to train.
The most straightforward solution is the straightthrough estimator (Bengio et al., 2013; Yin et al., 2019), which simply copies gradients across nondifferentiable layers, treating the Jacobian matrix as an identity matrix. SPIGOT (Peng et al., 2018; Mihaylova et al., 2020) further introduces a structured projection to reduce gradient errors. Building upon this previous work, VQ-VAE (van den Oord et al., 2017; Razavi et al., 2019) allows interleaving a discrete layer with continuous inputs mapped as discrete codes. These returned informative codes preserve necessary information for downstream tasks, and at the same time, provide an interface for controlling and interpreting the hidden states in intermediate layers of neural networks.
Another line of research turns to relaxation techniques, as they approximate discrete representation with continuous relaxation to mitigate the intractability issue. The Gumbel-softmax estimator (Jang et al., 2017) was first proposed by using the Gumbel-max trick (Maddison et al., 2014, 2017) to provide a differentiable approximation of the argmax operation. The marginal approaches (Friesen and Domingos, 2016; Kim et al., 2017; Liu and Lapata, 2018; Eisner, 2016) even further relax the discreteness restriction, not limiting themselves to selecting only a single discrete token, but instead aggregating information from all discrete tokens according to softmax distribution.
Different from all of them, our binary represen-
tation lies between continuous and discrete representations. On the one hand, the easy-to-train property of continuous representation allows us to avoid designing complex gradient estimators while also benefiting from state-of-the-art representation learning methods, i.e., contrastive learning. On the other hand, the easy-to-discretize property allows us to instantly transform these continuous representations into discrete representations, thus making the imposition of various task-relevant structures tractable and efficient. Besides, in other potential applications such as machine translation and language models, using binary representation removes the need for a large softmax, thus, reduces memory consumption and accelerates training and inference. Additionally, the final step of re-interpreting binary codes as labels or tokens allows the model to preserve much more contextual information within the codes. Consequently, this implicit clustering capability may potentially address issues like polysemy through discovering subclasses (§4.4).

heading: 6 Conclusions
text: In this paper, we demonstrated that introducing the binary representation to the output side is also feasible. Binary representation inherits the advantages of easy training from contiguous representation and the structure modeling capability from discrete representation. Therefore, binary representation can be considered as a novel representation that further bridges the gap between the continuous nature of deep learning and the discrete intrinsic property of natural languages. We achieved this by extending previous contrastive hashing into structured contrastive hashing. Specifically, we designed a bit-level CKY, defined a similarity function based on marginal probability, and proposed a novel contrastive hashing loss to mitigate the geometric center issue. Experiments show that our methods perform remarkably on various structured prediction tasks by using only a few bits. Our model also demonstrates a certain degree of implicit clustering capability in discovering subclasses. Most importantly, although we primarily focus on structured prediction tasks, our method can be easily applied to other natural language processing tasks by equipping the corresponding structures.
Limitations
Our contrastive hashing method yields informative binary representations. Nonetheless, information
theory suggests that labels with different frequencies carry varying amounts of information. Therefore, compressing all continuous label representation into codes with fixed number of bits is not the optimal solution, and it may even lead to the problem of high-frequency labels being overly concentrated in a narrow area. We remain to solve this issue by extending our method to a variant number of bits in future work.

heading: A Dataset Statistics

heading: Future Work
text: We remain to solve this issue by extending our method to a variant number of bits in future work