SlowGuess/ACL_2020_OCR huggingface dataset:
Bryan Klimt and Yiming Yang. 2004. The enron corpus: A new dataset for email classification research. In European Conference on Machine Learning, pages 217-226. Springer.
Andrew Lampert, Robert Dale, and Cecile Paris. 2010. Detecting emails containing requests for action. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 984-992. Association for Computational Linguistics.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Chu-Cheng Lin, Dongyeop Kang, Michael Gamon, and Patrick Pantel. 2018. Actionable email intent modeling with reparametrized rnns. In Thirty-Second AAAI Conference on Artificial Intelligence.
Douglas Oard, William Webber, David Kirsch, and Sergey Golitsynski. 2015. Avocado research email collection. In LDC2015T03. DVD. Philadelphia: Linguistic Data Consortium.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Sara Radicati and J Levenstein. 2015. Email statistics report, 2015-2019. Radicati Group, Palo Alto, CA, USA, Tech. Rep.
Owen Rambow, Lokesh Shrestha, John Chen, and Chirsty Lauridsen. 2004. Summarizing email threads. In Proceedings of HLT-NAACL 2004: Short Papers, HLT-NAACL-Short '04. Association for Computational Linguistics.
Simon Scerri, Gerhard Gossen, Brian Davis, and Siegfried Handschuh. 2010. Classifying action items for semantic email. In LREC.
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.
I Sutskever, O Vinyals, and QV Le. 2014. Sequence to sequence learning with neural networks. Advances in NIPS.
Wei Wang, Saghar Hosseini, Ahmed Hassan Awadallah, Paul N. Bennett, and Chris Quirk. 2019. Context-aware intent identification in email conversations. In Proceedings of the 42Nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'19.
Rui Zhang and Joel R. Tetreault. 2019. This email could save your life: Introducing the task of email subject line generation. In ACL.
A.1 Hyper-parameters
We now provide the hyper-parameters and training details for ease of reproducibility of our results. The encoder-decode architecture consists of LSTM units. The word embedding look-up matrix is initialized using Glove embeddings and then trained jointly to adapt to the structure of the problem. We found this step crucial for improved performance. Using random initialization or static Glove embeddings degraded performance.
We also experimented with using either a shared or a separate vocabulary for the encoder and decoder. A token was included in the vocabulary if it occurred at least 2 times in the training input/target. Separate vocabulary for source and target had better performance. Typically, source vocabulary had higher number of tokens than target. A shared dictionary led to increased number of parameters in the decoder and to subsequent over-fitting. The validation data was used for early stopping. The patience was decreased whenever either the validation token accuracy or perplexity failed to improve. We used the OpenNMT framework in PyTorch for all our Seq2Seq experiments.
Table 5 lists the hyper-parameters of the best performing model.
Hyper-parameter	Value
Rnn-type	LSTM
Rnn-size	256
# Layers	1
Word-embedding	100
Embedding init.	Glove
Batch size	64
Optimizer	Adagrad
Learning rate	0.15
Adagrad accumulator init.	0.1
Max. Gradient norm	2.0
Dropout	0.5
Attention dropout	0.5
Tokenizer	spacy
Vocabulary	Separate
Early Stopping (Patience)	5
Beam width	5
Table 5: Seq2Seq with copy mechanism : Hyperparameters for the best model.
A.2 Illustrative Examples
In this Section, we provide further examples of the email threads along with the highlighted commitment sentence. Note that some of the emails have previous thread email present, and some do not have it. For each of these examples, we also provide the To-Do item written by the human judge (denoted as GOLD) and that predicted by our best model (denoted as PRED). As in the main text, the sentences have been paraphrased and names changed due to the data sensitivity of Avocado.
8686
From: Beverly Evans To: Carlos Simmons Subject: Amazon.com update
Carlos, I came to know today from John Carter than we received a PHP script that is not decoding the correct database. Can you check with them why they sent us the eCommerce PHP code when the loss of functionality was not out fault? I have registered the error log in the eCommerce section because the staff scientist from Amazon mentioned it in his email. He also said they have not been able to resolve the issue and surprisingly did not mention who we should contact next. (This email exchange was about a week ago when I had handed them the cloud expenditures.) Also, we need to generate a PHP example to replicate the error. Could you update me if the team is working on it?
Thanks, Beverly
From: Carlos Simmons To: Beverly Evans Subject: Amazon.com update
The PHP they shared with us is an example. eCommerce is not what they want us to resolve. I feel we should wait until their engineers test all possibilities. Joseph informed us that they need to test the database more carefully and figure out which PHP code to send to us and whether they want our feedback on the database. I am not sure why they sent me a 'relevant PHP example' - I thought there was the only file they sent us yesterday. I will forward that to you and Renata.
GOLD: Forward PHP example to Beverly and Renata.
PRED: Forward eCommerce PHP to Beverly.
Table 6: Illustrative Example 1
From: Kirstin Barnes To: Nannie Jacobs Subject: Ready for Product Launch Nannie, I am ready for the product launch. I need to include some of the enhancements in the presentation. I'll submit what is already completed and then do the remaining after the meeting.. Kirstin Barnes Product Engineer AvocadoIT, Inc.
GOLD: Submit presentation with product enhancements.
PRED: Submit the enhancements for product launch.
Table 7: Illustrative Example 2
From: Rishabh Iyer To: R&D Subject: Software not ready yet for deployment
Hello, Unlike our plan last month, the software is still not ready for deployment. The team put together some errors last week. We must plan to make it available latest by next week. I will keep you posted.
Thanks, Rishabh Iyer.
Software Engineer AvocadoIT Inc.
GOLD: Keep r&d posted about deployment of software.
PRED: Keep r&d posted about deployment.
Table 8: Illustrative Example 3
From: Justine Sparrow To: Roma Patterson Subject: 24x7 Helpline
Roma,
I will bring this up in the Staff meeting today. I'll let you know the outcome. Could you confirm if this is for a license agreement or a shared solution ?
Thanks, Justine.
GOLD: Let Roma know result.
PRED: Let Roma know about the license agreement.
Table 9: Illustrative Example 4
8687
From: Rebecca Anderson To: Julia Roberts Subject: Run a bash script while synchronize Julia, When synchronizing is done, we want to run a bash script to delete old records on the machine and remove all activity logs. How can I do this? What is the way to perform this operation? Also, in the bash script, is there a way to sort the dates so that we can identify older activities? Thanks, Rebecca.
From: Julia Roberts To: Rebecca Anderson Subject: Run a bash script while synchronize Rebecca, We had exactly the same feature to delete activities which you mentioned in our previous release. But we no longer have that in the new version due to resource constraints. I will take to John to review this again. Thanks, Julia.
GOLD: Talk to John to review bash script again.
PRED: Talk to John to review the activities.
Table 10: Illustrative Example 5
From: Ramesh Paul To: Gopal Majumdar Subject: Updates List for 3/11 Here's the update for this week. 1. The R&D team is working on a presentation for the knowledge tranfer for v5. It should be ready within next two weeks. 2. I have received their email, but need to review the ppt. 3. Did you want to know more about the new cloud feature for automatic version management ? Or was it a different feature ? 4. I am constantly working on this. 5. Didn't we discuss this point in our last email ? 6. We are making similar tests in the desktop for v5 before migrating to the cloud. We first have to make sure things work well for the desktop. I will send you more details soon. Did you get a chance to update your blog with information about these new features ? Thanks, Ramesh.
GOLD: Send Gopal more details about tests in the desktop for v5.
PRED: Send Gopal more details on presentation for the knowledge transfer.
Table 11: Illustrative Example 6
From: Lori Howard To: Karen James; Bruce Thomas; Steve Perry Subject: Room reservations
Team,
This needs to be done through a formal training session, but as of now let me point out some crucial points about room reservations. 1. In case you allocate a room for general meetings and administrative work, then make sure you book it for that month, but not for long periods of time. (Karen, can you check with Renata whether this is fulfilled for our meetings next week?) 2. In case of clients who do not need the entire month, make sure to reserve only for the particular month. If it exceeds that time, the system will automatically resolve it and reserve it for next month. 3. For room reservation, either enter the number of hours required or the % of month, but not both. I would prefer precise hours. I will inform you when we can provide training, perhaps we can next week.
Thanks, Lori.
GOLD: Let Karen know about the training provide for room reservations.
PRED: Let Karen know about room reservations.
Table 12: Illustrative Example 7
8688
From: Matthew White To: Frank; Paul; Dennis Subject: Draft Agenda for Software Training
Dear All,
As discussed before, we have finally come to a concrete plan. I have attached the draft for your review. Please go over it and let me know asap your suggestions so that I can send them to the organizers. Please check the agenda and the names of trainees. I'll put together the Training plan and the overall 5-day agenda as soon as I can.
Matthew.
GOLD: Put together the training plan and the overall day agenda of software training.
PRED: Put together the draft agenda for software training.
Table 13: Illustrative Example 8
From: Diana Wilson To: Alba Deacon Subject: DHL package from IBM
Alba, I was able to track the package and as per the website it was in Sao Luis, Brazil at noon. I am not sure where it is, but it is Brazil so ... Send me an update if you receive it from them.
I just tracked the package and as of 10:00am today it was in Toluca, Mexico. Where that is I have no idea but it is in Mexico so ... Let me know if you hear from them when they receive it.
Thanks. Diana Wilson.
From: Alba Deacon To: Diana Wilson Subject: DHL package from IBM
Thanks Diana. If I hear anything I'll let you know.
Alba.
GOLD: Let Diana know about DHL package from IBM.
PRED: Let Diana know about DHL package from IBM.
Table 14: Illustrative Example 9