SlowGuess/NAACL_2021_OCR huggingface dataset:
Figure 5: Average BLEU score difference between classes having longer (more than 200 (median) words) and shorter descriptions larger (more than 200 (median) key-phrases) and shorter global schema for Ubuntu dataset. Lower difference indicates more invariance towards information available.
C More Experimental Analysis
We additionally report Krippendorff's alpha, a measure of annotator agreement for our human evaluation, on Amazon dataset. They are: for fluency 0.408, for relevancy 0.177, for missinginfo 0.226, and for usefulness 0.0948. For usefulness, we observe, if the systems are more distinct (GAN-Utility vs BART+missinfo+PPLM), then the agreement is higher i.e. 0.163. For missinginfo, again, 3-way gives higher agreement (0.434), and a probable cause would be that more annotations are going into the undecided category.
Additionally, Figure 5 shows the BLEU difference across different data samples (based on context length and global schema size) that follow a similar trend to Amazon. Table 9 shows generations from all the models, with a case the our best model trades off with missing information to improve the usefulness.
$^{14}$ https://huggingface.co/transformers/model_doc/bart.html  
 $^{15}$ https://github.com/uber-research/PPLM  
 $^{16}$ In comparison, humans get an F1-score of  $82.7\%$  in Amazon dataset
Amazon
Category	Bookshelf Speakers
Title	Yamaha NS-6490 3-Way Bookshelf Speakers Finish (Pair) Black
Description	Upgrade your current 5.1 home theater to a 7.1-Channel surround sound system by adding a pair of Yamaha NS-6490 bookshelf speakers. This speaker was designed for both professional & home entertainment enthusiasts with the capability to deliver a full, clear,...
Missing Schema	{speaker wire, mounting, (amplifier, tune, nsubj), wireless, bass, (iPhone, connect, obj), ...}
GAN-Utility	are these speakers compatible with a yamaha satellite speakers?
BART	What are the dimensions?
BART+missinfo	Do the speakers come with speaker wire?
BART+missinfo+PPLM	What kind mounting does this speaker use?
Category	Camera & Photo
Title	Porta Trace 10 x 12-inches Stainless Steel Frame Lightbox with Two 5000K Lamps
Description	Gagne Porta-Trace light boxes virtually eliminate the hot spots found in competitive lightbox units. Redesigned frame and reflector combine with the thick Plexiglas top to provide uniform and even lighting over the entire durable, stable viewing surface.
Missing Schema	Durable Stainless Steel frame will maintain its attractive appearance for years...
{camera, battery powered, flash, wireless, canon, nikon, ...}
GAN-Utility	will this work with a canon rebel?
BART	Does it come with the bulbs?
BART+missinfo	Is it battery powered?
BART+missinfo+PPLM	Can I replace the bulbs?
Table 9: Model generations for two examples product from Amazon. In the second example, our best model trades off with missing information to make the question more useful.