Title: TOWARD CONTROLLABLE TEXT CONTENT MANIPU-LATION

Abstract: Controlled generation of text is of high practical use. Recent efforts have made impressive progress in generating or editing sentences with given textual attributes (e.g., sentiment). This work studies a new practical setting of text content manipulation. Given a structured record, such as (PLAYER: Lebron, POINTS: 20, ASSISTS: 10), and a reference sentence, such as Kobe easily dropped 30 points, we aim to generate a sentence that accurately describes the full content in the record, with the same writing style (e.g., wording, transitions) of the reference. The problem combines the characteristics of data-to-text generation and style transfer, and is challenging to minimally yet effectively manipulate the text (by rewriting/adding/deleting text portions) to ensure fidelity to the structured content. We derive two datasets from the data-to-text task as our testbed, and develop a neural method with weakly supervised competing objectives and explicit content coverage constraints. Automatic and human evaluations show superiority of our approach over competitive methods including a strong rule-based baseline and prior approaches designed for style transfer.

1. INTRODUCTION:

Though significant progress of natural language generation has been made in recent years (Sutskever et al., 2014; Vaswani et al., 2017) , which is further boosted by the large scale model pre-training technique (Radford et al., 2018) , generating natural language with controllability is still under investigated. Controllability with existing models is mainly achieved by conditioning the generation on given inputs, which does not guarantee controll In this paper, we investigate controlled generation text generation problem in the context of language generation of structured information. Generating natural language text to describe structured content, such as a database record or a table, is of ubiquitous use in real-life applications including data report generation (Wiseman et al., 2017) , article writing (Lebret et al., 2016; Kiddon et al., 2016) , dialog systems (Wen et al., 2015; Yang et al., 2016) , and many others. Recent efforts have developed many techniques to improve fidelity to the source content, such as new powerful neural architectures (Gu et al., 2016; See et al., 2017) , hybrid generation and retrieval (Hashimoto et al., 2018; Weston et al., 2018) , and so forth, most of which are applied in supervised context.

Language is rich with variation-given a data record, there are diverse possible ways of saying the same content1 , with different word choices, expressions, transitions, tone, etc2 . Previous data-totext work has largely focused only on content fidelity, while ignoring and lacking control over the rich stylistic properties of language. It can be practically useful to generate text that is not only describing the conditioning content, but also following a designated writing style.

In this work, we study the new yet practical problem in which we aim to express given content with a sentence and mimic the writing style of a reference sentence (Fig 1 ). More specifically, we are given a structured data record containing the content to describe, along with a sentence about a similar but different matter. Our goal is to generate a new sentence that precisely depicts all content in the record, while at the same time using as much of the writing style of reference sentence as possible. As above, the problem differs critically from the supervised data-to-text (Wiseman et al., Paul Millsap was close behind with 10 points ( 4 -7 FG ) , 12 rebounds and 8 assists in 29 minutes . Located near Ranch, you can find a family-friendly place named Cotto serving French food rated 5 stars .

Domain NBA Reports Restaurant Reviews:

Iman Shumpert was close behind with 15 points ( 6 -15 FG ) and 10 rebounds in 38 minutes .

Located near Clare Hall , you can find a place at riverside named Loch Fyne serving Italian food priced £20-25 .

Content:

Reference Figure 1 : An example input (content record and reference sentence) of text content manipulation andits desired output. Text portions that fulfill the writing style are highlighted in right.

2017) or retrieval-and-rewriting work (Hashimoto et al., 2018; Weston et al., 2018) as we have imposed an additional goal of preserving the reference text style. The resulting problem is typically unsupervised due to lack of parallel data.

The problem also differs in important ways from the emerging task of text style transfer (Hu et al., 2017; Shen et al., 2017) which assumes an existing sentence of certain content, and modifies single or multiple textual attributes of the sentence (e.g., transferring negative sentiment to positive) without changing the content. Our task, on the contrary, assumes abstract style is encoded in a reference sentence and attempts to modify its concrete content to express new information from the structured record. The different setting can lead to different application scenarios in practice, and pose unique technical challenges. In particular, though the most recent style transfer research (Subramanian et al., 2019; Logeswaran et al., 2018) has controlled multiple categorical attributes which are largely independent or loosely correlated to each other, a content record in our task, in comparison, can contain varying number of entries which are of different types (e.g., player, points, defensive/offensive rebounds, etc), having many possible values (e.g., hundreds of players), and are structurally coupled (e.g., 32 points by Lebron). A model must understand the content structure, and minimally yet sufficiently manipulate the reference sentence by rewriting, adding, or deleting text portions, with necessary polishing for grammatical correctness and fluency. We name the problem text content manipulation. Our empirical studies show the most recent models designed for style transfer fail to perform well in the task.

In this paper, we first develop a large unsupervised dataset as a testbed of the new task. The dataset is derived from an NBA game report corpus (Wiseman et al., 2017) . In each data instance, besides a content record and a reference sentence as the problem inputs, we also collect side information useful for unsupervised learning. Specifically, each instance has an auxiliary sentence that was originally written by human reporters to describe the content record without seeing (and thus stylistically irrelevant to) the reference sentence. We also provide the structured record of the reference sentence. The side information can provide valuable clues for models to understand the content structure and text semantics at training time. We do not rely on the side information at test time.

We then propose a neural method to tackle the problem. With a hybrid attention and copy mechanism, the model effectively encodes the reference and faithfully copies content from the record. The model is learned with two competing objectives of reconstructing the auxiliary sentence (for content fidelity) and the reference sentence (for style preservation). We further improve the model with an explicit content coverage constraint which encourages to precisely and fully convey the structured content.

For the empirical study, we devise automatic metrics to measure content fidelity and style preservation, respectively. We also perform human evaluations to compare different approaches. Results demonstrate the proposed method significantly improves over others, including a strong rule-based baseline and the recent style transfer models.

2. RELATED WORK:

Generating text conditioning on structured input has been widely studied in recent work (Wen et al., 2015; Lebret et al., 2016; Yang et al., 2016; Wiseman et al., 2017, etc) . Those methods are based on neural sequence to sequence models and trained with supervised data. This line of work has focused primarily on generating more accurate description of the given data, while it has not studied the problem of controlling the writing style of outputs. Our task takes a step forward to simultaneously describing desired content and controlling stylistic properties. Furthermore, our task is challenging due to its unsupervised setting in practice.

Beyond generating text from scratch, there is another line of work that first retrieves a similar sentence and then rewrites it to express desired information (Hashimoto et al., 2018; Weston et al., 2018; Li et al., 2018; Guu et al., 2018) . For example, Weston et al. (2018) Recently, there has been growing interest in text style transfer, in which many techniques for controlled text generation are developed (Hu et al., 2017; Shen et al., 2017; Yang et al., 2018; Prabhumoye et al., 2018; Tian et al., 2018; Subramanian et al., 2019; Logeswaran et al., 2018) Both (Subramanian et al., 2019) and (Logeswaran et al., 2018) generalized the task to controlling multiple categorical attributes at the same time. Our work differs from those in that we assume an existing sentence to provide the source of style and a structured record as the source of content.

The input content record in our task is also more structured than the style attributes which are typically loosely connected and of a pre-fixed number. The resulting content manipulation setting poses unique challenges in controlling, as discussed more in the empirical study.

3. TASK DEFINITION: TEXT CONTENT MANIPULATION:

We first formally define the problem of unsupervised text content manipulation, and establish the notations. We then illustrate the generality of the task via comparing with two exiting tasks.

Without loss of generality, consider a content record x = {x i } Lx i=1 , where each element x i is a data tuple which typically includes a data type (e.g., points), a value (e.g., 32), and other information (such as the associated player, e.g., Lebron James). L x is the number of tuples in record x, which can vary across different records. We are also given a reference sentence y which is assumed to describe content that has a similar but not exact the same structure with that of the record x. For example, in Fig 1, both the content record and the reference sentence involve two players, respectively, but the number of associated data tuples as well as the types are different (e.g., Lebron James in the record has 3 box-score entries, while Jrue Holiday in the reference has only 2).

We may also have access to other side information at training time. For example, in the dataset developed below, each content record x is associated with an auxiliary sentence y that was originally written to describe x without following the reference y . Each reference sentence y also has its corresponding record x containing the content information. The side information can provide valuable clues for models to understand the content structure and text semantics at training time. For example, the auxiliary sentence provides a hint on how the desired content can be presented in natural language, though it is stylistically irrelevant to the reference sentence. Note that, at test time, a solution to the task should only rely on the inputs (x, y ) without using the side information.

The goal of the task is to generate a new realistic sentence ŷ that achieves (1) content fidelity by accurately describing the full content in x, and at the same time (2) style preservation by retaining as much of the writing style and characteristics of reference y as possible. The task is unsupervised as there is no ground-truth sentence for training.

4. PROPOSED MODEL:

We next develop methods to tackle the problem. As shown in the empirical study (section 6), a simple rule-based method that matches x with (x , y ) and performs text replacement would fail in terms of content fidelity due to the different structures between x and x . Previous approaches for (multi-attribute) style transfer do not apply well either, because of the different underlying task assumptions and the rich content structures of records with varying lengths.

In the following, we present a new neural approach that addresses the challenges of text content manipulation. We first describe the model architecture, then develop unsupervised learning objectives, and finally add a content coverage constraint to improve learning. Figure 2 provides an illustration of the proposed approach.

Let p θ (ŷ|x, y ) denote the model that takes in a record x and a reference sentence y , and generates an output sentence ŷ. Here θ is the model parameter.

4.1. HYBRID ATTENTION-COPY MECHANISM:

As shown in the figure, the architecture of our neural model consists of two encoders and one decoder. The first encoder is used to extract representation of the reference sentence y . The second encoder is applied to content records. Specifically, for each data tuple in a record, we first concatenate the embedding vectors of all fields in the tuple, and feed the combined embedding to the encoder. There is no need to specify a particular order of the tuples as their fields have specified the associated player or team.

The decoder is to generate the output sentence, with a hybrid attention-copy mechanism at each decoding step. In particular, the decoder applies a joint attention (Luong et al., 2015) over both y and x, and applies a copy mechanism (Gu et al., 2016) only on the record x. More specifically, at each step t, the decoder first attends jointly to the hidden states of both encoders and obtains a decoding hidden state h t . The decoder then computes the output distribution over words with

EQUATION

where g t is the probability of generating a token from the vocabulary; p

(t)

V is the generation distribution over the whole vocabulary; while p (t)

x is the copy distribution over the data values in the content record. All the quantities are computed based on the post-attention hidden state h t .

4.2. COMPETING LEARNING OBJECTIVES:

As defined in section 3, the task has two simultaneous goals, namely data fidelity and style preservation. The two goals are in a sense competitive with each other (e.g., describing the new designated content would usually change the expressions in reference sentence to some extent). We base our unsupervised learning on this competitive relation.

We make use of the side information (y , x ) during training. Specifically, as the auxiliary sentence y was originally written by human to describe the content x and thus can be seen to have the maximum data fidelity, we devise the first objective that reconstructs y given (x, y ):

Lcontent(θ) = log p θ (y | x, y ).

(2)

We call it the content fidelity objective.

To fulfill the second goal of preserving the style of reference y , we want to encourage the model to generate sentences in a similar form of y . We further notice that, if we feed the model with the reference sentence y and its corresponding record x (instead of x), the ground truth output of the case is indeed y itself (as y describes content x , and is of course in the same style of itself). We thus can specify the second objective that reconstructs y given (x , y ):

EQUATION

We call it the style preservation objective. The objective essentially treats the reference sentence encoder and the decoder together as an auto-encoding module, and effectively drives the model to absorb the characteristics of reference sentence and apply to the generated one.

The above two objectives are coupled together and train the model to achieve the desired goals:

EQUATION

)

where λ is the balancing parameter.

4.3. CONTENT COVERAGE CONSTRAINT:

As shown in the empirical study (section 6), the above learning can yield reasonably good performance, but sometimes can still fall short of accurately expressing the desired content. We thus devise an additional learning constraint based on the nature of content description-each data tuple in the content record should usually be mentioned exactly once in the generated sentence.

The copy mechanism over content record x enables a simple yet effective way to encourage the behavior. Intuitively, we want each tuple to be copied once and only once on average. We thus minimize the following L2 constraint that drives the aggregated copy probability of each data tuple to be 1:

EQUATION

where p

(t)

x , as defined in Eq.equation 1, denotes the copy distribution over all data tuples at decoding step t; and 1 is a vector with all ones. It is still possible that tokens of the content values "leak" from the generation distribution p (t)

V in Eq.equation 1. We disable the leakage by masking out relevant words (particularly numbers) for each instance from the vocabulary.

We note that prior work in other context, especially machine translation, has also explored the idea of coverage through either architecture augmentation (Tu et al., 2016) or inference penalty (Wu et al., 2016) . We tried these techniques but did not obtain noticeable improvement. As shown in the experiments, the proposed explicit coverage constraint over copy mechanism leads to significant performance gains.

The full training objective of the proposed model with the constraint is thus written as:

EQUATION

)

where η is the weight of the constraint.

5. DATASET:

We now present two datasets developed for the task3 . The first dataset is derived from a recent large table-to-document corpus (Wiseman et al., 2017) supervised game report generation which has attracted increasing research interest (Nie et al., 2018; Wiseman et al., 2017) . The second dataset is derived from the E2E Generation dataset (Dušek et al., 2019) , which is a crowd-sourced dataset in the restaurant domain.

NBA News Reports To obtain our NBA data, we first split each game report into individual sentences, and, for each sentence, find its corresponding data in the box-score table as the content record. A record can contain a varying number of tuples, with each tuple containing three fields, namely a data type, a value, and an associated player or team, e.g., (team points, 106, Lakers). As the original corpus is already largely clean, we found some simple rules are sufficient to obtain high-quality results in this step. Please see the supplementary materials for more details. Each of the resulting record-sentence pairs is treated as a pair of (x, y ) , namely (content record, auxiliary sentence). The next step is to find a suitable reference sentence y for each content record x. As defined above, the reference sentence should cover similar but not the same content as in record x.

We achieve this by retrieving from the data another record-sentence pair using x, where the retrieved record is designated to have a slightly different structure than that of x by having less or more tuples and different data types. More details of the retrieval method are deferred to supplements. The retrieved record-sentence pair thus plays the role of (x , y ) and is paired with (x, y ) to form an instance.

Restaurant Reviews riginal E2E dataset consists of the review records, e.g.(eatType, coffee shop, Bibimbap House) and natural language descriptions of restaurants. It enables us to explicitly construct the record-sentence pairs (x, y ) . We construct this data set in a similar way to the NBA new reports data set. In contrast, to make this task more challenging, we adopt a new retrieval method in constructing y , that is, we constraint the corresponding x to have at at least 2 different types of entries compared with x.

6. EXPERIMENTS:

We conduct both automatic and human evaluations to assess the model performance. For automatic evaluation, we use two metrics to measure content fidelity and style preservation, respectively. Results show our model balances well between the two goals, and outperforms a variety of comparison methods. All code will be released soon.

6.1. EXPERIMENTAL SETUP:



Comparison Approaches:

We compare with a diverse set of approaches:

• AttnCopy-S2S. We first evaluate a base sequence-to-sequence (Sutskever et al., 2014 ) model with the above attention-copy mechanism, which takes in record x and generates its descriptive sentence y. The evaluation provides a sense of the difficulty in describing desired content. • Rule-based Method. A straightforward way for text content manipulation is to match between

x, x and y with certain rules, and replace corresponding portions in y with those in x. Specifically, we first build a mapping between the tuples of x and x through their data types, and a mapping between x and y through data values, types and indicative tokens (e.g., "12 points" in y indicates 12 is of type player points or team points). The two mappings connect x and y , enabling us to swap appropriate text in y to express content x.

In theory, rule-based method sets the best possible style preservation performance, as it only replaces content related tokens (particularly numbers) without modifying other parts of the reference sentence. The output, however, tends to miss or contain extra content compared to the content record of interest.

• Multi-Attribute Style Transfer (MAST) (Subramanian et al., 2019) . We compare with the most recent style transfer approach that models multiple attributes. To apply to our setting, we treat content record x as the attributes. The method is based on back-translation (Sennrich et al., 2015) that first generates a target sentence ŷ conditioning on (x, y ), and then treat it as the reference to reconstruct y conditioning on (x , ŷ). Auxiliary sentence y is used in an extra auto-encoding loss.

• Adversarial Style Transfer (AdvST) (Logeswaran et al., 2018) . As another latest style transfer approach capable of handling more than one attributes, the model also mixes back-translation with auto-encoding as the above method, and additionally uses adversarial training to disentangle content and style representations.

• Ours LSTM w/o Coverage. For ablation study, we compare with our LSTM model variant that omits the content coverage constraint. That is, the model is trained by maximizing only Eq.equation 4.

• Ours Transformer w/o Copy. We empirically found that disenabling the self-attention to y only at the final decoding block can achieve the similiar copying effects. The results are resonable , since this operation enables it attending to x to the greatest extent. Hence, we compare with this kind of standard transformer model.

• Ours Transformer w/o Coverage. For ablation study, we compare with our transformer model variant that omits the content coverage constraint . That is, the model is trained by maximizing only Eq.equation 4.

Model Configurations:

We employ experiments on both LSTM and Transformer. For LSTM, we used are single-layer encoder-decoder architecture with the Luong attention (Luong et al., 2015) . For Transformer, we set the number of blocks in encoders and decoders to be 3. For both models, we employ the copy mechanism Vinyals et al. (2015) to improve performance. Both the embedding dimension and hidden dimension are set to 384. During training, we first set (λ = 0, η = 0) to pre-train the model to convergence so that the model captures the full characteristics of the reference sentence. Then we switch to set (λ = 0.2, η = 1.0) for full training. We apply Adam optimization (Kingma & Ba, 2014) with an initial learning rate of 0.001 and gradient norm clipping of 15. For inference we use beam search with beam-width 5. The maximum decoding length is set to 50.

6.2. AUTOMATIC EVALUATION:

As no ground truth annotations are available, we first set up automatic metrics for quantitatively measuring the key aspects of model performance.

Metrics:

We propose metrics to evaluate the two primary goals of the task, namely content fidelity and style preservation. A desired solution should balance and excel on both metrics.

• Content fidelity. For the NBA dataset, we use an information extraction (IE) approach to measure content fidelity following the data-to-text task (Wiseman et al., 2017) . That is, given a generated sentence ŷ and the conditioning content record x, we extract data tuples from ŷ with an IE tool, and compute the precision and recall against x. We use the IE model provided in (Wiseman et al., 2017) , which achieves around 81% precision and 86% recall on the test set. For the Restaurant dataset, we train a binary transformer classifier to evaluate whether ŷ manipulate the content successfully or not. The classifier achieves 94% accuracy on the test set. Specifically, we define the content fidelity score as the accuracy of overall new content records of x that are expressed by ŷ. The content residual score is the percentage of old content records that should be deleted in ŷ. The desired model should express the more new content records and less old ones. Hence, lower residual score represent the stronger ability of manipulating the old content. Table 2 : Model performance under automatic evaluation. Results are averaged over 3 runs ± one standard deviation. Except for the content score 2, the higher the score is, the better performance is. Models in the first block (AttnCopy Seq2seq and Rule-based) represent two baselines for reference performance. We have highlighted the best results in blocks 2, 3 and 4 under different metrics. Our two models with LSTM or Transformer achieve significant higher precision and recall respectively compared to both rule-based and style transfer methods, and reaches a high BLEU score in style preservation.

• Style preservation. A generated sentence is desired to retain stylistic properties, such as word choice and expressions, of the input reference sentence. Inspired by the text style transfer literature (Yang et al., 2018; Subramanian et al., 2019) , we measure the BLEU score between generated and reference sentences. To reduce the influence of new content, we first mask in both sentences all obvious content tokens, including player/team names and numbers, by replacing them with a special token <M>, and then compute the BLEU score. In this way, the above rule-based method has a maximum BLEU score of 100, which is consistent with our intuition above.

Results:

We now compare the performance of different methods in terms of the above metrics. Table 2 shows the quantitative results obtained on NBA and Restaurant datasets.

The first block shows the two baseline models providing reference performance. The AttnCopy-S2S model only concerns about content fidelity, and achieves a high content precision score (but a low recall). However, its style BLEU is particularly low, which verifies the rich variation in language and that direct supervised learning is incapable of controlling the variation. As discussed above, the rule-based method can reach the maximum BLEU (100) after masking out content tokens while its content precision score is unsatisfactory. The two style transfer methods (MAST and AdvST) fail the expectation, as their content fidelity performance is greatly inferior or merely comparable to the rule-based method. This is partially because these models are built on a different task assumption (i.e., modifying independent textual attributes) and cannot manipulate content well.

For the NBA dataset, our proposed LSTM/Transformer models achieve better content precision/recall, substantially improving over other methods (e.g., with a 12-point precision boost in comparison with the rule-based baseline) except for AttnCopy-S2S which has failed in style control.

Our methods also manage to preserve a high BLEU score of over 80. The superior performance of the full model on both datasets compared to the variant Ours-w/o-Coverage demonstrates the usefulness of the content coverage constraint (Eq.5). By explicitly encouraging the model to mention each of the data tuples exactly once-a common pattern of human-written descriptions-the model achieves higher content fidelity with less style-preservation ability "sacrificed". We see similar trend for the Restaurant dataset, our proposed LSTM/Transformer based models achieve better fidelity and residual score. One of the reason why our proposed Transformer-based model is worse than LSTM-based model may be owing to the fact that restaurant dataset has shorter sentence length and this may limit the power of Transformer in capturing the long semantic dependency. Humans are asked to score the model outputs in terms of content fidelity, style preservation, and fluecny, respectively, from 1 (strongly bad) to 5 (strongly good). As expected, the rule-based method reaches the maximum possible scores in terms of style preservation and fluency, but a much lower score in terms of content fidelity. Our LSTM-based model is more balanced across all aspects, and performs significantly better in accurately describing desired content. Bottom: Humans are asked to rank a pair of generated sentences in which one is from our model and the other from the comparison method. Our model wins on more than 50% instances compared to each of other models on two datasets.

6.3. HUMAN EVALUATION:

We also carried out human evaluation for a more thorough and accurate comparison. Following the experimental settings in prior work (Subramanian et al., 2019; Logeswaran et al., 2018; Shen et al., 2017) , we undertook two types of human studies: (1) We asked three human turkers to directly score generated sentences in three aspects, namely content fidelity, style preservation, and sentence fluency. Each score is from 1 (strongly bad) to 5 (strongly good); (2) We present to annotators a pair of generated sentences, one from our model and the other from a comparison method, then ask the annotators to rank the two sentences by considering all the criteria. Annotators can also choose "no preference" if the sentences are equally good or bad. For each study, we evaluate on 80 test instances, and compare our model with the rule-based method, AdvST style transfer model (which has shown better performance on the task than the other style transfer model MAST), and the model variant without coverage constraint.

Table 3 shows the human evaluation results. From the top block of the table, as expected and discussed above, the rule-based method sets the records of style preservation and fluency scores, as it only conducts lightweight token replacement on reference sentences. However, its content fidelity score is very low. In contrast, our model achieves a reasonably high content score of 3.88 and 4.08, which is much higher than those of other methods. The model is also more balanced across the three criteria, achieving reasonably high scores in both style preservation and language fluency. The fluency of the full model is slightly inferior to the variant without coverage constraint, which is not unexpected since the full model has modified more portions of reference sentence in order to better describe the desired content, which would tend to introduce more language mistakes as well.

The bottom block of Table 3 shows the results of ranking sentence pairs. We can see that our model consistently outperforms the comparison methods with over 50% wins.

6.4. QUALITATIVE STUDY:

Table 4 shows example outputs on two test cases given content record x and reference sentence y . We can see that, in general, the proposed full model can manipulate the reference sentence more accurately to express the new content. For example, in the first case, the rule-based method was confused between the winning and losing teams, due to its incapacity of understanding the semantics of text such as "held off". The style transfer model AdvST failed to comprehend the content record well and generated irrelevant data "100 -100". The simplified variant without explicit coverage constraint copied the content of Bulls twice. In contrast, the full model successfully generates the desired sentence.

7. DISCUSSIONS:

We have proposed a new and practical task of text content manipulation which aims to generate a sentence that describes desired content from a structured record (content fidelity) and meanwhile follows the writing style of a reference sentence (style preservation). To study the unsupervised problem, we derived a new dataset, and developed a method with competing learning objectives and an explicit coverage constraint. For empirical study, we devised two automatic metrics to measure different aspects of model performance. Both automatic and human evaluations showed superiority of the proposed approach.

There are multiple directions to further improve and generalize the work, including enhancing text generation quality in general with more sophisticated neural architectures (e.g., Vaswani et al., 2017) and learning algorithms (e.g., Tan et al., 2018; Rennie et al., 2017) , incorporating richer problem structures (e.g., structures between x and x ) and linguistic knowledge through learning constraints (Hu et al., 2018; 2016) and structure bias (Strubell et al., 2018; Yang et al., 2016) , as well as better leveraging the reference sentence as a template through mask-and-infilling (Zhu et al., 2019) (much like the rule-based approach). It is also interesting to extend from generating a single sentence to generating a whole passage (e.g., a game report or a news article) with any desired writing style (Wiseman et al., 2017) .

It is difficult to mathematically define text style, content, and the boundary between them. In the task of text style transfer, a style (or attribute) has to be explicitly defined (usually as a categorical variable), which can be difficult when it comes to abstract properties (e.g., word choices). In comparison, our setting of defining writing style based on a reference sentence provides an alternative, arguably more natural way of specifying the style of interest. This resembles the style definition in image style transfer (Gatys et al., 2016; Johnson et al., 2016) where visual style is specified with a reference stylistic image.

Besides the direct practical use of the task itself, the objective of preserving writing characteristics of a given reference sentence provides a way of preventing a data-to-text model from repeatedly generating generic, low-diversity text (Guu et al., 2017; Li et al., 2016) . Generation diversity is explicitly controlled by the richness of reference sentences.

A DATA CREATION:

Our dataset is made from the ROTOWIRE dataset (Wiseman et al., 2017) , which consists of paired tables and paragraphs, each describing a NBA game. Each table is consisted of data records about box-and line-scores of basketball games, while each corresponding paragraph written by specialists is summarizing the game. In order to obtain desired (x, y) pairs, we first separate the paragraphs into sentences. Each sentence will be our y. However, it is not so easy to obtain the corresponding x (i.e. data records): we have to find out which data records in the table of data records are described in y.

To achieve this, we proposed a not precise but accurate enough rule based method to get x from y. Two steps are performed in our method:

Step 1, find out all entities (player, team, and city) and numbers in y;

Step 2, pick out candidate (entity, number) pairs that will form our final x.

In step 1, we first collect all team/player/city names as all entities in the training set. We locate these entities in y and replace every multi-word entity name in y by an underscore-connected single token in order to simplify the task. These tokens are then recognized as entities. To find out the numbers, we simply invokes a Python module called text2num (Copyed from exogen's text2num.py) which can recognize and convert every English numbers in y to digital numbers. Then all digital numbers are recognized.

In step 2, we enumerate every pair of (entity, number) found in step 1 and retrieve all candidate data records from the table. We simply iterate over the table and pick out those records whose entity name and score are exactly the same as the entity and number in our pair. However, there can be multiple records picked out which have different or even contradict labels due to the ambiguity. To reduce wrong records retrieved, we add more rule constraints to filter out obviously wrong records. For instance, if the succeeding token of the number is 'assist', then those records with labels related to 'rebound' or 'turnover' are obviously wrong. After this, there is still some redundancy in these records, though, we believe it is accurate enough to collect these records as x. Finally, the entity names are also added to x so that the copy mechanism can directly copy the entity names from x.

After obtained all (x, y) pairs of the training/validation/test sets, we now have to assign a (x , y ) to each of them. The (x , y ) is always from the training set, even for (x, y) in the validation/test sets.

The (x , y ) should meet our requirement that x and x are quite similar but not completely match in their labels. We define a match function to calculate the similarity between x and all x in the training set. The match function is: we first define a similarity score between every pair of records between x and x , then use Kuhn-Munkres algorithm to find out the best match of the bipartite graph with scores as weights. The sum of scores in the match is our similarity of x and x , which is the output of our function. The xx match also act as an crucial part in our rule based model described above.

:

https://en.wikipedia.org/wiki/Variation_(linguistics)

We refer to these characteristics that are independent of the desired content as writing style.

The dataset is released at https://github.com/ZhitingHu/text_content_manipulation

Figures and Tables:

Figure 2: A (simplified) data example (left) and the model overview (right).

The data statistics for two different datasets, which both derived from the exiting data-totext datasets.

Table content: <table><tr><td/><td colspan="2">NBA Reports</td><td colspan="3">Restaurant Reviews</td></tr><tr><td>Train</td><td>Dev</td><td>Test</td><td>Train</td><td>Dev</td><td>Test</td></tr><tr><td colspan="2">#Instances 31,444 6,765</td><td>6,930</td><td colspan="2">29,486 6,299</td><td>6,273</td></tr><tr><td colspan="6">#Tokens 7.88M 1.69M 1.75M 0.54M 0.12M 0.12M</td></tr><tr><td>Avg Sentence Length 25.07</td><td>25.10</td><td>25.32</td><td>18.36</td><td>18.34</td><td>18.35</td></tr><tr><td>#Data Types 34</td><td>34</td><td>34</td><td>8</td><td>8</td><td>8</td></tr><tr><td>Avg Record Length 4.32</td><td>4.31</td><td>4.35</td><td>5.38</td><td>5.38</td><td>5.35</td></tr></table>

Human Evaluation Results. Top:

Table content: <table/>

The Pistons ( 22 -33 ) held off the Bulls ( 34 -21 ) 100 -91 in Detroit on Friday night . Rule-based The Mavericks ( 30 -14 ) held off the Bulls ( 29 -16 ) 98 -102 in Dallas on Friday night .

Table content: <table><tr><td>Content x</td><td>TEAM WINS LOSSES TEAM-PTS TEAM Bulls 29 16 102 Mavericks 30 WINS LOSSES TEAM-PTS TEAM 14 98 Dallas</td></tr><tr><td>Reference y</td><td/></tr><tr><td colspan="2">AdvST The Bulls ( 29 -16 ) held off the Mavericks ( 30 -14 ) 100 -100 in Dallas on Friday night .</td></tr><tr><td colspan="2">Ours w/o Cover. The Bulls ( 29 -16 ) held off the Bulls ( 29 -16 ) 102 -98 in Dallas on Friday night .</td></tr><tr><td colspan="2">Ours The Bulls ( 29 -16 ) held off the Mavericks ( 30 -14 ) 102 -98 in Dallas on Friday night .</td></tr></table>

Example outputs. Text of erroneous content is highlighted in red, where [...] indicates desired content is missing. Text portions in the reference sentences and the generated sentences by our model that fulfill the stylistic characteristics are highlighted in blue. Please see the text for more details.

Table content: <table/>

Peer review (iclr2020):
This paper addresses an interesting case of text generation, interpolating between the requirements of content fidelity and style transfer. This is carried out using a hybrid *attention-copy* mechanism, which is intended to attend to style in a reference sentence and copy content over from a structured data representation. This along with data augmentation and a combined unsupervised objective comprise the learning framework. An ablation study and human evaluation are carried out, confirming that the model presented has advantages over foregoing style transfer and generation models. I think this paper should be accepted as it addresses an interesting and widely applicable scenario, and the combined unsupervised objective is a creative and well-motivated solution. It consitutes a contribution both to neural generation and style transfer methods. It is not a strong accept as I think that when introducing a new task, more in-depth evaluation and error analysis is appropriate. One challenge with this task is evaluation -- some of the scores are close enough to 50% that it would be useful to have significance of some kind calculated, more information about the annotators and inter annotator agreement, and some amount of error analysis to help give context for Table 3. Particularly, since this is a new task, the foregoing models compared would be expected to perform worse than the new model. To learn more about ways in which the new model is well adapted to the task, it could be useful to look closer into the tradeoff between fidelity and style transfer, e.g. comapring the BLEU score between y and y_hat in addition to y* and y_hat for all models. Recommendations: 1. If possible, please use clearer notation to explain the data augmentation step. It is really interesting and creative, but very hard to follow x, x*, y, y* -- please consider using subscripts like y_side or something. 2. In section 6.2, a binary transformer classifier is mentioned. Please clarify how *manipulating the content successfully* is measured, and describe the classification task in further detail.