Year: 2021

Abstract:
Existing methods for style transfer operate either with paired sentences or distributionally matched corpora which differ only in the desired style. In this paper, we relax this restriction and consider data sources with additional confounding differences, from which the desired style needs to be inferred. Specifically, we first learn an invariant style classifier that takes out nuisance variation, and then introduce an orthogonal classifier that highlights the confounding cues. The resulting pair of classifiers guide us to transfer text in the specified direction, creating sentences of the type not seen during training. Experiments show that using positive and negative review datasets from different categories, we can successfully transfer the sentiment without changing the category.1

heading: 1 INTRODUCTION
text: Despite advances in neural text generation (Brown et al., 2020), fine-grained control over generated outputs remains challenging. Indeed, the ability to easily transfer output style by altering attributes such as sentiment, formality, genre, and personal styles would make text generation tools more appealing (Hu et al., 2017; Shen et al., 2017; Rao & Tetreault, 2018; Xu et al., 2012).
Solving style transfer typically requires the method to be able to disentangle what should be transferred from orthogonal aspects of sentences that ought to be kept intact. This disentanglement problem can be largely avoided in simple supervised scenarios with access to parallel sentences differing only in style (e.g., sentiment transfer with parallel negative and positive sentences, Figure 1.a). Recent approaches address a more difficult version of the task by dispensing with parallel sentences (Shen et al., 2017). Nevertheless, they assume that the corpora differing in style remain distributionally matched in other ways (e.g., sentiment transfer with negative and positive reviews from the same category of products, Figure 1.b).
However, data available for training style transfer models is rarely distributionally matched, and often involves changes other than style as well (e.g., sentiment transfer where negative and positive reviews come from different product categories, Figure 1.c). This means that the desired style difference is no longer illustrated directly as the difference between the two corpora, making the task substantially more challenging. More subtly, the model is asked to generate sentences it has not seen during training, since the generated sentences in a new style should not also reproduce those confounding differences present in the training data.
Solving style transfer with confounding cues requires us to also learn what the desired style difference is. This can be facilitated by dividing the data into two groups of different styles, while the sets within each group illustrate variation we do not wish to alter. In the example in Figure 1.c, reviews are divided into two groups according to their sentiment. Within a group, each dataset corresponds to a different product category which should be preserved. Besides sentiment transfer, this setup easily generalizes to other style transfer tasks, e.g., dialectal transfer. In this case, the groups will be divided according to the dialects, and sets within each group will represent different speakers whose personal style should be preserved.
Our model builds on invariant risk minimization (Arjovsky et al., 2019) to infer the style as an invariant distinction across different datasets from the two groups. The resulting style classifier leaves complementary aspects of sentences to be controlled (preserved). We can illustrate aspects orthogonal to style with new sets of environments and learn another invariant classifier. Together,
1Our code will be publicly released after the review process.
the two classifiers are used to guide sentence generation in a style transfer model along the desired direction. Combining with back-translation (Sennrich et al., 2015; Zhu et al., 2017) and language model regularization (He et al., 2020) techniques, we can generate new types of sentences that have not been seen during training.
We empirically evaluate our proposed model in two sentiment transfer settings that involve confounding factors. In the first setting, we augment original review data with special tokens, creating a spurious correlation with sentiment. In the second setting, we consider sentiment transfer where negative and positive reviews come from non-overlapping product categories. In both cases, we assess the ability of the model to transfer the sentiment, while preserving other aspects – special tokens in the first case, and product category in the second case. Our experiments demonstrate that our model successfully achieves both tasks, bringing significant gains over baselines that do not consider confounding factors. For instance, on the the task of sentiment transfer from different product categories, the model yields 28.4% increase in category preservation; according to human evaluation, its success rate is 6.2% higher than the best previous system.

heading: 2 RELATED WORK
text: The task of style transfer is related to paraphrasing, whose goal is to generate multiple linguistic realizations of the same underlying content (Androutsopoulos & Malakasiotis, 2010). However, style transfer adds an additional complexity – the requirement to control for a specific realization characteristic during rewriting. While paraphrasing models have been used in the past for the task of style transfer, these models are impacted by arbitrary variations present in paraphrasing datasets (Preotiuc-Pietro et al., 2016; Gröndahl & Asokan, 2019; Krishna et al., 2020). We instead take a data-driven approach for discerning style from content, where the style is not strictly limited to paraphrasing and can involve more general attribute transfer such as sentiment and political slant transfer (Prabhumoye et al., 2018).
Recent work in non-parallel style transfer proposes different techniques such as cross-alignment (Shen et al., 2017), delete and retrieval (Li et al., 2018), or parallel latent sequences (He et al., 2020). The generation of these approaches all takes place “in-distribution”, i.e., realizing sentences of the type already seen during training. The sentences we generate are by design not seen during training (Figure 1.c). Subramanian et al. (2018) proposed multi-attribute style transfer that controls both the sentiment and the category of a sentence. However, similar to other prior work, they assumed access to all sentiment-category combinations for training. In contrast, we assume that negative and positive sentences come from non-overlapping categories.
Since we have to infer style from datasets with confounding cues, we make use of invariant learning (Peters et al., 2016; Arjovsky et al., 2019) to estimate style as an invariant direction. Moreover, our classifiers need to perform well across different combinations of source/target datasets and generalize
to generated text (out-of-distribution samples) to properly guide the transfer model. Therefore, our problem is also related to domain adaptation (Ben-David et al., 2010; Ganin & Lempitsky, 2015; Ganin et al., 2016) and domain generalization (Blanchard et al., 2011; Muandet et al., 2013). Our main contribution is the utilization of invariance for guiding the generation process.

heading: 3 STYLE TRANSFER WITH CONFOUNDERS
text: We consider two groups of datasets: GA = {A1, . . . , An} where each Ai (i = 1, . . . , n) is a dataset consisting of sentences with style sA (e.g., negative sentiment), and GB = {Bn+1, . . . , Bn+m} where each Bj (j = n+ 1, . . . , n+m) similarly conforms to style sB (e.g., positive sentiment). In addition to style, a dataset has its own characteristics different from each other (e.g., category). Our goal is to transfer a sentence x of style sA into style sB (and vice versa) without changing its content or other characteristics.
One attempt is to aggregate collections of sentences in each group A = A1 ∪ · · · ∪ An, B = Bn+1 ∪ · · · ∪Bn+m and perform style transfer between them. However, the specific characteristics of Ai and Bj will become confounding factors and will be changed along with style. Instead, we notice that style is an invariant distinction between group GA and group GB . In other words, the style difference is stable across different Ai and Bj . Therefore, we can learn to isolate it by taking out intra-group variations. Once we have access to style, we can learn to transfer sentences along this direction while preserving other sentence characteristics.
We take a two-step procedure to accomplish this task. We first learn a pair of invariant classifiers to detect style and orthogonal characteristics. Then we use the classifiers to guide the learning of a style transfer model. Each part of the model is described in detail below.

heading: 3.1 INVARIANT CLASSIFIERS
text: We make use of Invariant Risk Minimization (Arjovsky et al., 2019, IRM) to learn our style and orthogonal classifiers. IRM requires us to specify a set of environments E = {e1, . . . , eE}, where each e ∈ E represents data {(xei , yei )} ne i=1 collected under a certain environment. The different environments account for nuisance variation that the classifier should not pay attention to. The IRM objective is to learn a feature representation that enables a classifier to be simultaneously optimal for all environments. The rationale is that such representation likely involves primarily causal features that remain stable regardless of the nuisance variation. Therefore, the classifier can better generalize to new, unseen test environments compared to the standard Empirical Risk Minimization (ERM) classifier trained on the pooled data from all environments.
We adopt the IRMv1 formulation, which does not explicitly separate the representation from the classifier but treats the classifier output itself as the representation. In this vein, the objective becomes to minimize the classifier loss across all the data while penalizing per-environment gradients with respect to any multiplier of the classifier output:
min Φ:X→Y ∑ e∈E Re(Φ) + λ‖∇w|w=1.0Re(w · Φ)‖2 (1)
where Re(f) := EXe,Y e [`(f(Xe), Y e)] is the risk of f under environment e (` can be any loss function), and λ is a hyperparameter weighting the gradient penalty term. Gradients would be zero if Φ is per-environment optimal. It remains to define suitable environments for the invariant classifiers to separate style and confounding factors in our task.

heading: 3.1.1 STYLE CLASSIFIER
text: To learn a classifier to distinguish between styles sA and sB without relying the specific characteristics of each dataset, we pair each Ai and each Bj to form environments: ei,j = {(x, y = 0) | x ∈ Ai} ∪ {(x, y = 1) | x ∈ Bj} (2) and learn an IRM classifier Cs : X → Y to predict the group label across environments {ei,j}1≤i≤n,n+1≤j≤n+m (cf. Figure 2.a). Since all Ai datasets share style sA, and all Bj datasets share style sB , style is a feature representation that elicits an invariant classifier across different environments. Conversely, if the classifier uses any features specific to Ai/Bj , it will not be optimal in another environment consisting of a different pair Ai′/Bj′ , thus violating the IRM constraint.

heading: 3.1.2 ORTHOGONAL CLASSIFIER
text: In addition to requiring the transferred output to have a different style from the input, we also require it to retain the other characteristics of the input. To detect style-independent characteristics, we construct environments based on predictions of the style classifier Cs. These style-dependent environments serve to illustrate to the second invariant classifier Co what it should not rely on.
Let D = {(x, y = 0) | x ∈ A} ∪ {(x, y = 1) | x ∈ B} be the entire dataset, aggregate of all corpora. We create two environments2:
e1 = {(x, y) ∈ D | Cs(y|x) > 0.5} (3) e2 = {(x, y) ∈ D | Cs(y|x) ≤ 0.5}
and learn an IRM classifier Co : X → Y across {e1, e2}. In this way, Co cannot depend on the direction quantified by Cs (i.e., the inferred style), and must find other orthogonal features to distinguish A from B (cf. Figure 2.b).
Note that if we have successfully transferred a sentence in Ai to style sB , then Cs should predict label 1 for the output sentence because its style has been changed, while Co should continue to assign label 0 as the orthogonal characteristics ought to have remained intact.

heading: 3.2 STYLE TRANSFER MODEL
text: Based on the pair of invariant classifiers, we can now build a style transfer model to transfer a sentence to a different style specified by Cs while preserving other characteristics controlled by Co. Formally, we have dataset D = {(x, y)}, where y denotes the group label of sentence x. We learn a style transfer model M : X × Y → X that takes a source sentence x and a target group y as input, and outputs a revised sentence that conforms to the style of group y. The model is learned first in a reconstruction phase and then in a transfer phase, discussed in turn.
When the input sentence x is from group y, M should behave as an autoencoder and reconstruct x. Therefore, we have the reconstruction loss:
Lrec(θM ;x, y) = − log pM (x|x, y) (4)
where θM denotes the parameters of M to be learned. We first use Lrec to train M on D for an epoch to provide the model with a good initialization to generate realistic sentences.
After initializing M as an autoencoder, we use the pair of invariant classifiers to guide it towards appropriate style transfer. Given an example (x, y) in D, we let x̃ ∼ pM (·|x, 1−y) be the transferred output sampled from the model. If successfully transferred, x̃ should have style different from x according to the style classifier Cs, and all the other characteristics should be the same as x according
2A set of more segmented environments can be created based on the confidence of Cs. However, we did not observe performance gains with more environments.
to the orthogonal classifier Co. Namely, Cs should treat x̃ as coming from a different group, and Co should treat x̃ as coming from the same group. These two constraints lead to losses:
LCs(θM ; y, x̃) = − log pCs(1− y|x̃) (5) LCo(θM ; y, x̃) = − log pCo(y|x̃) (6)
We further use a pre-trained language model L and introduce a KL divergence term DKL(pM (·|x, 1− y)‖pL) to regularize the transfer distribution. Estimating the KL divergence with one sample of x̃ ∼ pM (·|x, 1− y), we have:
LLM(θM ;x, y, x̃) = − log pL(x̃) + log pM (x̃|x, 1− y) (7)
The first term ensures the fluency of the generated sentence x̃. The second term corresponds to the negative entropy of the transfer distribution −HpM (·|x,1−y). Maximizing this entropy term encourages exploration and helps to avoid bad local optima of constantly generating similar sentences (He et al., 2020).
Finally, we include the back-translation loss that the original sentence x should be generated if we take x̃ as input and set the target label to be y (Sennrich et al., 2015; Zhu et al., 2017):
LBT(θM ;x, y, x̃) = − log pM (x|x̃, y) (8)
Taken together, our overall training objective is:
E(x,y)∼D,x̃∼pM (·|x,1−y)[λ1LCs + λ2LCo + λ3LLM + λ4LBT] (9)
In practice, we use word-level loss for LLM and LBT (i.e., divided by the sentence length) to make them comparable in magnitude to LCs and LCo . We use Gumbel-Softmax (Jang et al., 2016) to approximate the discrete sampling process of x̃ to compute gradients. At test time, we perform greedy decoding to generate the transferred output.

heading: 4 EXPERIMENTS
text: To assess the ability of our model to perform style transfer in the presence of confounders, we consider two experimental settings. In the first experiment, we compose a synthetic task by modifying sentence punctuation to create a spurious correlation with sentiment (Choe et al., 2020). The model needs to transfer the sentiment while preserving the punctuation. In the second experiment, we consider sentiment transfer across different product categories. The goal is to transfer sentiment without changing the product category.
Baselines We consider two variants of our full model: the first is guided by the style classifier Cs alone without the orthogonal classifier Co; the second is guided by an ERM classifier CERM trained on D, performing direct transfer between A and B without taking into account confounders. We also compare with He et al. (2020), whose training objective has a back-translation loss and a KL divergence term similar to ours. They do not use a classifier, but rely on two language models separately trained on A and B to promote the transferred sentence to have the target style. Finally, we compare with a paraphrasing based method of Krishna et al. (2020), which again transfers between the aggregated A and B without considering confounding factors.
Model Architecture We implement the classifiers Cs, Co using TextCNN (Kim, 2014) with sliding windows over 3-5 words. On the synthetic task, however, we found that because the dataset is simple to classify, the training loss of the CNN classifier is close to 0. This removes any guidance from IRM constraints, degenerating the resulting classifier to a standard ERM version. Therefore, to ensure that we estimate invariant classifiers, we use less powerful Bag-of-Words classifiers in the synthetic task. The employed language model L is a 1-layer LSTM trained on A ∪ B. The style transfer model M consists of a 1-layer LSTM encoder and a 1-layer LSTM decoder augmented with attention mechanism (Bahdanau et al., 2014). To achieve style control through y, we learn two style embeddings for y = 0, y = 1 respectively, and add the corresponding one to each word embedding before feeding to the decoder.
Training Regime and Hyperparameters During the training of the invariant classifiers, we linearly anneal the gradient penalty coefficient λ from 0 to λmax over the first 100K steps, and continue training with λ = λmax for another 200K steps. We try λmax in {1000, 3000, 5000} and choose the one with the best performance on the validation set. In the training of the style transfer model M , we linearly anneal the temperature of Gumbel-Softmax from 1 to 0.1 in the first 20K steps, and then keep it at 0.1 in the next 10K steps. We try the weights of the loss terms {λ1, λ2, λ3, λ4} in {1, 2, 4}, and select a model that strikes a good balance between output perplexity, BLEU with input, and accuracies according to the style and orthogonal classifiers. We note that different weight combinations will lead to different trade-offs (Pang, 2019), and we provide more detailed results in Appendix B.

heading: 4.1 SENTIMENT TRANSFER WITH DIFFERENT PUNCTUATION
text: We aim to emulate the presence of confounding factors by modifying sentences in a standard sentiment transfer dataset with special symbols. For a sentence x, we remove its original punctuation, and then add either an exclamation mark “!” or a period “.” at the end. Let p be the probability of adding “!”, and 1− p be the probability of adding “.”. p varies across different corpora. This setting enables us to measure model’s ability to transfer sentence sentiment without changing its punctuation.
Dataset We adapt the sentiment transfer dataset introduced by Shen et al. (2017), which has 177K negative sentences and 267K positive sentences for training, 2K negative and positive sentences for validation, and 500 negative and positive sentences for testing. We obtain new training, validation and test sets using the following procedure: (1) negative sentences are modified with p = 0 to form A1; (2) positive sentences are equally divided into two sets, and modified with p = 1, p = 0.8 to form B2, B3 respectively. By construction, the punctuation strongly correlates with the sentiment. Therefore, a direct transfer between A1 and B2 ∪B3 is likely to change the punctuation as well. By observing that sentiment is invariant while p is variant in B2 and B3, we aim to transfer only the sentiment, not the punctuation.
Evaluation We assess transfer accuracy by comparing change in sentiment in the input and output sentences. To automate this evaluation, we utilize a separate sentiment classifier trained on negative and positive sentences both modified with p = 0.5. In this way, its prediction will not be affected by punctuation. Note that this classifier is only used for evaluation, not for training the style transfer model. We also evaluate model’s ability to keep punctuation intact during transfer by directly comparing input and output punctuation. To measure fluency, we report the perplexity of the output measured by an unbiased language model trained on sentences modified with p = 0.5. Finally, we compute the BLEU score of the output with respect to a human reference (Li et al., 2018).
Results We first report performance of the invariant classifiers Cs and Co in the setting reflecting their intended use in our style transfer model. We reverse the correlation between punctuation and sentiment labels to simulate style transferred sentences, and assess their ability to predict the desired aspects. Specifically, we test the accuracy of classifiers on positive sentences modified with p = 0 and negative sentences modified with p = 1, p = 0.8. Table 1 shows results of Cs and Co compared with the standard CERM classifier trained on D. As expected, CERM performs poorly, mixing sentiment and punctuation clues. In contrast, the invariant classifier successfully separating the two aspects, achieving accuracy of 75.3% on sentiment prediction and 89.9% on punctuation prediction.
Table 2 summarizes style transfer results. The paraphrasing-based method of Krishna et al. (2020) is not suitable for sentiment transfer that requires semantic changes, as shown by its low sentiment accuracy of 28.4%. Moreover, it has the lowest BLEU score because it introduces unnecessary rewriting learned from the paraphrasing dataset. Despite reaching high sentiment accuracy, the direct
transfer methods of He et al. (2020) and M guided by CERM have low punctuation accuracy (4.6% and 4.5%, respectively). Our full model achieves both high sentiment accuracy (84.3%) and high punctuation accuracy (97.7%). The reliance on the orthogonal classifier Co is proved critical – its omission results in dramatic drop of punctuation accuracy (5.5%). Output examples in Table 3 provide multiple illustrations of these phenomena: Krishna et al. (2020) often paraphrases the input without changing the sentiment, while other models consistently change punctuation; only our full model successfully transfers the sentiment without changing the punctuation.

heading: 4.2 SENTIMENT TRANSFER WITH DIFFERENT CATEGORIES
text: Our second experiment focuses on sentiment transfer with product category as a confounding factor. Previous work took negative and positive reviews from the same category (Li et al., 2018). Our setting is more challenging where negative and positive reviews belong to distinct, non-overlapping categories. The model needs to infer that it is the sentiment to be transferred, not the category.
Dataset We use the 5-core Amazon review data (Ni et al., 2019), focusing on four large product categories: Clothing Shoes and Jewelry (CSJ), Home and Kitchen (HK), Electronics (E), and Cell Phone and Accessories (CPA). We further filter reviews based on their length, keeping reviews of length 5-20 words. Following standard practice, reviews with rating above three are considered positive, and those below three are considered negative. Reviews with rating three are discarded. Specifically, A1 consists of negative reviews from the CSJ category, A2 - negative reviews from HK, B3 - positive reviews from E, and B4 - positive from CPA. We create a dataset of 150K sentences for each category, in which 130K are used for training, 10K for validation and 10K for testing. Note that we do not use any review data of different sentiments from the same category.
Evaluations To assess sentiment transfer accuracy automatically, we train a classifier on complete data which includes negative and positive reviews from all four categories. It has test accuracy of 96.0%. Moreover, we train a 4-way category classifier using complete data to assess whether the output preserves input product category. It has test accuracy3 71.0%. As in the previous task, we report the perplexity of the output measured by an unbiased language model trained on the complete data. Note that these omniscient models are used only for evaluation and not for training the style transfer model. Since we do not have human references for this task, we compute the BLEU score with respect to source sentences as a rough indicator of content preservation (Zhao et al., 2018). In addition, we conduct human evaluation by asking human judges to rate each output on a Likert scale from 1 to 5 on three criteria: sentiment transfer, content preservation, and fluency. We consider a generated output successful if it is rated 4 or 5 on all three criteria. We evaluated 200 randomly sampled examples (100 negative and 100 positive) and collected two annotations for each sentence.
Results Table 4 shows automatic evaluation results for different models. Our model yields significant performance gains over the baselines, achieving 57.3% absolute increase in sentiment transfer compared to Krishna et al. (2020), and 28.4% increase in category preservation compared to He et al. (2020). The human evaluation results in Table 5 further verify the superiority of our model. Krishna et al. (2020) has the highest score in fluency as their model is fine-tuned from GPT2-Large. Our model achieves the highest sentiment transfer score and the highest success rate.
3This accuracy can be attributed to the fact that some short sentences do not have clear indicators of their product category.

heading: 5 CONCLUSION
text: In this paper, we consider a new, challenging version of the style transfer task, where the model has to exclude confounders and infer the desired transfer direction from data. We propose to learn a pair of invariant classifiers to detect style and orthogonal characteristics, and then use them to guide a style transfer model. We empirically demonstrate significant performance gains over direct style transfer in two experimental settings. While this technology shows significant promise, it still requires further development to be used in practice. However, as style transfer algorithms continue to advance, their ability to support rapid, at scale content modification will increase. This can potentially result in dissemination of fake news and other forms of misinformation.

heading: A CLASSIFIER ACCURACY IN THE REAL TASK
text: Here, we report performance of the invariant classifiers Cs and Co to classify based on sentiment and category respectively when their coupling is reversed (Section 4.2). We take positive reviews from CSJ and HK, and take negative reviews from E and CPA. The test accuracy in shown in Table 7.

heading: B TRADE-OFF OF DIFFERENT WEIGHT COMBINATIONS OF LOSS TERMS

heading: C ADDITIONAL EXAMPLES

Review summaries:
This paper considers a text style transfer task where no paired sentences are available. For example, we have positive Clothing reviews and negative Cell Phone reviews, the paper tries to generate negative Clothing reviews. Note that it is likely the model transfers not only the sentiment (from positive to negative) but also product category (from clothing to cell phone). To avoid that, the paper proposes to learn an invariant style classifier (which can classify the sentiment regardless of the product category). Besides, the paper also proposes to learn an orthogonal classifier which can monitor any style-independent changes (e.g. product category). 

This paper presents a method that can work on a single text style transfer task with multiple datasets. One example scenario as exemplified in the paper is the sentiment transfer task with datasets on multiple product categories. The basic idea of this work is to design a cross alignment between datasets from different categories, so the learning algorithm can focus on the expected stylistic information and eliminate other confounding factors. The proposed methods were evaluated on the sentiment transfer task with two different setups to demonstrate the value. 

This article deals with style transfer. It proposes an approach to generate texts with opposite sentiments in a adversarial context where the style is correlated with other contextual factors such as domain or noisy punctuation. Given the fact that other contextual factors are weakly labeled, the authors introduce Invariant Classifers to extract the style from other attributes. The general framework has been defined by (Arjovsky et al., 2019, IRM) and the basic classifier is (Kim, 2014). Section 3.2 describes the authors' contribution where multiple orthogonal classifiers enables them to transfer style while preserving other attributes. The authors propose relevant baselines corresponding to hypotheses ablations. The text generator is a 1-layer LSTM. The authors demonstrate that their approach is able to transfer sentiment while preserving domain or punctuation depending on the experiment.

Style transfer is a text generation task where a certain attribute of a sentence (like formality) is modified while preserving sentence content. It is generally studied in "unsupervised" settings --- no access to parallel data of sentences differing only in the target attribute. Nevertheless, algorithms assume access to corpora of unpaired sentences in each of the attributes, which may significantly differ in content. Prior works [1, 2, 3, 5] have noted this difference in content, which is encouraging models to modify non-target attributes like semantics (See Table 1 in [4]).  This paper is an attempt to fix this issue using invariant risk minimization (IRM) [6]. IRM encourages classifiers to be invariant across "environments", or datasets which share a target class but differ in other aspects which are not essential for classification ("spurious correlations"). In this work two spurious correlations are studied in sentiment transfer --- (1) a synthetic punctuation correlation; (2) product category of Amazon reviews. To perform style transfer, the authors first train two orthogonal IRM classifiers --- the first classifies the target attribute ("sentiment") irrespective of the confounder ("product category"), and the second for the confounder ("product category") irrespective of sentiment. These two classifiers are used to guide model generation, along with an LM smoothing and backtranslation objective (similar to [4]).  The authors compare their proposed approach against strong baselines and report promising results.  [1] - https://arxiv.org/abs/1910.03747   [2] - https://arxiv.org/abs/2010.05700   [3] - https://dl.acm.org/doi/10.5555/3016100.3016326   [4] - https://arxiv.org/abs/1811.00552   [5] - https://arxiv.org/abs/1905.13464   [6] - https://arxiv.org/abs/1907.02893