メールサービスを面白くするためのAIと自然言語処理の基本と応用 。
Futuristic Communication
The Role of AI in Email Services

JPAAWG 6TH MEETING - 2023
1Consent.
❑❑❑

I warmly welcome attendees to share insights and learnings frommy presentation at JPAAWG 6th General Meeting on social media. I
suggest using the official hashtag when posting on platforms such
as Twitter, Facebook, and Instagram to help others easily find and
follow the conversation.
Photography and video recording are allowed during mypresentation, and I appreciate if you could attribute any shared
content to me and JPAAWG 6th General Meeting.
However, I ask that you respect the privacy and preferences ofothers in the audience. If someone indicates that they do not wish tobe photographed or recorded, please respect their wishes.
Thank you for your cooperation and understanding.
2

About Meoooooo

Name
: Nuwan Senevirathne (ヌワン)
Company : Qualitia
Position
: AI Engineer

I love gaming
League of Legend (LOL)
AD main

3

Agenda
01

02

Neural Networks.

Attention is All You Need.

03

04

Subject Line Generation.

Email Mis-sending Prevention.

4

01
Neural Networks.
Simple Introduction.

5

Neural Networks.
Perceptrons. [1957]

Feedforward Neural Network. [1960]

Recurrent Neural Networks. (RNNs) [1980]

Convolutional Neural Networks. (CNNs) [1990]

Long Short-Term Memory Networks. (LSTM) [1997]

Transformer. [2017]

Etc

https://www.researchgate.net/profile/Juxi-Leitner 6

02Attention is All You Need.
Transformer Model

7

What is Transformer.
o Neural Network Architecture.
o Vaswani et al. in the paper "Attention is All You Need” 2017.
o Based on attention mechanisms.
o Major component is muti-head self attention.
o Useful in;
▪ Machine translations.
▪ Text Generations.
▪ Text Classifications.

8

Components of the Transformer Model.
❑ Encoder
❖
❖
❖

Takes Input data.
Transform into series of numerical vectors.
Each vector capture the meaning of word in
the context of the whole sentence.

❑ Decoder
❖

❖

Takes the vectors.
Generate the output.

9

Attention Mechanism.
❑ General Concept
➢ Allows a model to focus on specific parts of theinput when producing an output.

➢ Weighs the importance of different parts of theinput.
➢ Decide how much focus to put on different areas.

❑ Types
➢ Scaled dot-product attention,
➢ Multiplicative attention
➢ Additive attention
➢ Etc.
10

Self Attention.
❑ Scaled dot-product attention.
❑ Specific Type of Attention:
o Model attends to all parts of theinput simultaneously
o Compute the representation of
each part in the context of all
other parts.

❑ Consider the entire context of a sentence.

11

Tranformer: Conclusion
❑

Innovative architecture.

❑Widely used in modern NLP.

❑Understand words

❑

Understand context.

❑Understand relationships.

❑❑

Many models followed this model architecture and produce SOTA
results on various of NLP tasks.
If anyone need more information, visit
https://jalammar.github.io/illustrated-transformer/ . This post
explain the things very well.
12

AI in Email Services and Security.
❑Spam Detection.

❑

Phishing Detection.

❑Malware Detection.

❑

Anomoly Detection.

❑

Risk Scoring.

❑

Data Loss Prevention.

❑

There are many more things.

13

03
Subject Line Generation.
Finetuned custom T5 Model.

14

What is Subject Line Generation?
Email Body

AI Model

Subject Lines

1.

User inputs email content.

2.

AI model analyzes the content.

3.

AI model generates a list of potential subject lines.

4.

More effective approach.

5.

Eliminates the need for users to brainstorm a subject line.

15

Subject Line Generation Example

1.
2.
3.
4.

AI model to a
More effective subject lines.
AI subject line generators first analyze the content of your email.
They then use this information to generate a list of potential subject lines.

16

Summarization Methods.
Source Document

Extractive Summary❑

Source Document

Extractive summarization.
o Indentify Key phrases or sentences
o Use those extracted phrases as summary.
o Same sentences and structutre.

Abstractive Summary❑Abstractive summarization.
o Paraphrasing.
o Different words, different sentence structure.
o Concise summary.

https://wandb.ai/events/summarization-trial-t5/reports/Model-Recycling-Flan-T5-and-Dialogue-Summarization--VmlldzozNjg5NTU517

Why Abstractive Summarization?
❖Short.

❖Concise.

❖Directly relevant to the contents.

❖Capture mainpoint or purpose.

❑Example: (Email content)
みなさま、お疲れ様です。平野です。
M3AAWGとは何か、M3AAWGでのメールセキュリティや周辺トピックの最新の話題、サンフランシスコの最近の雰囲気、などについてお話しします。
oo

❑

Subject: " M3AAWG概要とサンフランシスコ現地レポートのご案内".
18

T5 (Text-To-Text TransferTransformer) Model.
1.

Treats every NLP problem as a text-to-text problem.

2.

Causal Language Model.

3.

Heart of T5 are self-attention mechanisms.

4.

T5 uses Relative Positional Encoding..

5.

Causal masking.

6.

Encoder-decoder.

https://arxiv.org/pdf/1910.10683.pdf

19

Overview of the T5 process.

Input Text

Tokenization

Attention
Masks

Decoding

Beam
Search

Output

Encoding

20

T5: Text embedding flow.
Tokenization

Word Embeddings
Positional Encodings
Combination of Embeddings

Contextualized Embeddings
21

T5: Tokenization.
❑What is tokenization
❑❑

Split text into smaller units.

There are different ways to do tokenization.

[ Learning, is, fun ]
Word Tokenization

Learning is fun

[ L, e, a, r, n, i, n, g, _, i, s, _, f, u, n ]
Character Tokenization

[ Learn , ##ing, is, fun ]
T5 Tokenization (Sentencepiece)

22

T5: Tokenization.

Input text

T5 Tokenizer

Tokenized Text

Input text:

今日はいい天気ですね。外に出て散歩したいです。

▁今日 はいい天気ですね。

外に出て散歩したいです。 


54634

2090

18709

22034

429767

5823169

5440

876

4

7123Word Embeddings.
o 14 Tokens
o Each Token mapped to a vector size of 768
o 14 x 768 Vector

Tokenized Text

Word Embeddings

T5 Model▁

今日 はいい天気ですね。

外に出て散歩したいです。 


54634

2090

18709

22034

4

29767

58

23169

5440

876

4

7

Token Indexes

1

Word Embeddings

5-12.5625

12.0625

-14.0625

...

-0.2695

-5.7812

15.0625

4634

-3.5156

-7.3750

13.6875

...

12.4375

-10.6875

0.5312

7-13.3750

7.5000

-8.8125

...

3.3750

6.3125

10.1875

...

...

...

...

...

...

...

2.6250
-10.7500

12.1875
5.6875

-10.0000
-12.4375

...
...

-0.7773
29.1250

-9.7500
4.5625

8.0625
21.1250

…
41

Embedding

768

No. of
Tokens

24

Contextualized Embeddings
WordEmbeddings

Positional
Encodings

Combination ofEmbeddings

Contextualized
Embeddings

Word Embeddings

-12.5625

12.0625

-14.0625

...

-0.2695

-5.7812

15.0625

-3.5156

-7.3750

13.6875

...

12.4375

-10.6875

0.5312

-13.3750

7.5000

-8.8125

...

3.3750

6.3125

10.1875

...

...

...

...

...

...

...

2.6250
-10.7500

12.1875
5.6875

-10.0000
-12.4375...
...

-0.7773
29.1250

-9.7500
4.5625

8.0625
21.1250

Transformer
Layers

Contextualized Embeddings

-2.4007e-01

2.7876e-01

-3.2107e-01

...

-5.1860e-01

4.3266e-01

6.2798e-01

-8.6244e-02

...

-6.8169e-01

...

...

...

...

...

7.5833e-03

5.4663e-03

-7.6981e-03

...

-1.4981e-02

25

T5 Embeddings.
❑Encapsulate various aspects of the text's meaning.

❑Allowing the model to perform complex text-to-text tasks.

❑Form the basis of the model's ability to understand and generate text.

-2.4007e-01

2.7876e-01

-3.2107e-01

...

-5.1860e-01

4.3266e-01

6.2798e-01

-8.6244e-02

...

-6.8169e-01

...

...

...

...

...

7.5833e-03

5.4663e-03

-7.6981e-03

...

-1.4981e-02

26

Training Data.
❑Preprocess, clean.

❑No need to label.

❑❑

Subject

❑

Email Body

Emails from company employees.

27

Model Architecture.
Training Data

T5 Tokenizer

T5 Model

Style Layer

Output
28

What is Fine-tuning.
1.

Process of adapting a pre-trained model to specific task.

2.

Don’t need to trained from scratch.

3.

Efficient.

4.

Better performance.

5.

Flexible.

29

T5: Fine-tuning Process.

1.

Tokenization

2.

Model Architecture

3.

Pre-trained Weights

30

T5: Fine-tuning Process.
4.

Decoder Adaption

5.

Attention Masking

6.

Loss Function

7.

Fine-tuning Parameters

31

Accuracy Matrices.
❑

Many Evaluation Matrices.

❑Famous:
ROUGE Score
BLEU
METEOR
BERT Score
Etc
❑
❑
❑
❑

❑

❑

We Used BERT Score.

32

BERT Score.
❑Uses BERT embeddings for evaluation.

❑Overcomes n-gram metric limitations.

❑Measures: Precision, Recall, F1-Score.

❑Benefits:
Resilient to rephrasing.
Aligns with human judgments.
❑❑

❑

Applications: Summarization, Translation.
33

Our Results.

34

How We Improved the Model.
1.

Fine tuning gives us a good result.

2.

We added custom layers for the model.
❑Layer to generate a summary in a specific style.

3.

Language model to remove personal Information.

4.

Set the right hyperparameters.

5.

❑Learning rate

❑

Weight decay

❑

Optimizer

❑

Etc

Use a diverse dataset.

35

Final Pipeline.

Email Body

T5 Tokenizer

TokenizedInputs

Custom T5ModelSubjectLinesMaskPersonalInformation

NER Model

Style Layer

36

Subject Generation DEMO.

37

Subject Generation Demonstration.
❑ Actual Subject: インフルエンザ予防接種 詳細

38

Subject Generation Demonstration.
❑ Actual Subject: インフルエンザ予防接種 詳細
❑ Generated Subjects:
o 【インフルエンザ予防接種日程のご連絡
o 【人事総務部】インフルエンザ予防接種日程のご連絡
o インフルエンザ予防接種日程のご連絡(人事総務部)

39

04
Email Mis-sending Prevention
By Named Entity Recognition.

40

Email Mis-Sending !.
❑How ?
o

❑

Incorrect email address.

Damages.
o

Leaking personal information.

o

Severe consequences.

o

Leaking secrets.

o

Leaking sensitive information.

oDamage professional relationships.

41

Our Method.
Deep Learning Model
Receiver’s first name

Email Content

Extract Entities

Receiver’s last name
Receiver’s company
name

To Address

Company+Domain
DBEtc..

To Domain name

Compare

Extracted
Domain name

Mismatch

Warning !
42

Japanese E-mails.
Examples

43

Named Entity Recognition.
Taylor Swift is singing in a concert in Tokyo this coming December.
Person

Location

Time

❑

Task of identifying names in text and classifying into set of predefined catergories.

❑Three universally accepted catergories:
o❑

Common tasks:o❑

Person, Loaction and Organization.
Recognition of date, time, email address, etc

Domain specific entities (email):
oSender name, receiver name, address, phone number, etc
44

Named Entity Recognition.
株式会社クオリティアヌワン セネビラツナ様

45

Named Entity Recognition.
株式会社クオリティアヌワン セネビラツナ様
Tokenization

46

Named Entity Recognition.
株式会社クオリティアヌワン セネビラツナ様
Tokenization

株式会社

クオリティア

ヌワン

セネビラツナ

様

47

Named Entity Recognition.
株式会社クオリティアヌワン セネビラツナ様
Tokenization

株式会社

クオリティア

ヌワン

セネビラツナ

様

XLM-RoBERTa + softmax classifier

48

Named Entity Recognition.
株式会社クオリティアヌワン セネビラツナ様
Tokenization

株式会社

クオリティア

ヌワン

セネビラツナ

様

XLM-RoBERTa + softmax classifier

P(株式会社)

P(クオリティア)

Receiver’s company

Receiver’s company

P(ヌワン)

Receiver’s first name

P(セネビラツナ)

P(様)

Receiver’s first name
49

What is XLM-RoBERTa.
❑XLM-RoBERTa (Cross-lingual Model-RoBERTa).

❑RoBERTa, a robustly optimized BERT variant.

❑Supports 100+ languages.

❑

Transformer-based, like BERT.

❑Encoder Only.

❑12 encoder layers for base model.

50

Implement NER System.

❑Prepare training data
o

Collect Data

o

Filter and Select Data

o

Tag dataset.

oDoccanooo

https://github.com/doccano/doccano

26 Entities

51

Tagged Data.

52Fix Sequence Mismatch.
❑Aligning tagged label sequence to model Tokenization.
Entity Name

Short Name

Receiver’s company

CO_R

Receiver’s first name

FN_R

Receiver’s last name

LN_R

Other

O
CO_R

FN_R

クオリティア

ヌワン

CO_R

Taged Data

XLM-RoBERTa
Tokenization

株式会社
CO_R

CO_R

FN_R

LN_R

セネビラツナ
LN_R

LN_R

様

LN_R

株式会社 ク オ リティ ア ヌ ワン ▁ セ ネ ビ ラ ツ ナ 様
CO_R

CO_R

CO_R

FN_R

LN_R

LN_R

LN_R

53

Adjusting Sequence Mismatch.
Token

Named Entity

株式会社

CO_R

クオリティア

CO_R

ヌワン

FN_R
O

セネビラツナ
様a.

LN_R
O

After adjusting for sequence mismatch for using model tokenizer.

TokenNamed Entity

株式会社

CO_R

ク

CO_R

オ

CO_R

リティ

CO_R

ア

CO_R

ヌ

FN_R

ワン

FN_R

▁O

セ

LN_R

ネ

LN_R

ビ

LN_R

ラ

LN_R

ツ

LN_R

ナ

LN_R

様O
54

Model Architecture.
Training Data

XLM-RoBERTa Tokenizer
XLM-RoBERTa Model

Dropout Layer
Linear Layer

Softmax Function
Entities
55

Fine-tuning Process.
1.

Tokenization and Text Encoding.

2.

Add Special Tokens.

3.

Calculate Positional Embeddings.

4.

Load Pre-trained Weights.

56

Fine-tuning Process.
5.

Fine-tuning Layers

6.

Classification Head.

7.

Loss Function.

8.

Fine-tuning Parameters.

57

Model Results.

❑ High Precision.

❑ High Recall.
❑ High F1 Score.

58

Inference.

Email Body

Fine Tuned
Entity and probability
XLM – RoBERTa Model

Accept

Probability
Score ≥ 90%Reject

59

Receiver’s Company Name Extract DEMO.

60

Receiver’s Company Name Extract Demonstration.

Extracted company name
Actual company name

61

Challenges.
❑❑❑

❑

Insufficient training data.

Language and domain adaptation.
Annotator bias and inconsistency.

Costly to annotate data for domain specific text.

62

Conclusion.
❖Neural Networks.

❖Transformers Architecture.

❖Summarization Methods.

❖T5 Model.

❖Process of Building Subject Generation Model .

❖Named Entity Recognition.

❖XLM-RoBERTa Model.

❖Building Email Mis-sending prevention Model.
63

Thank You!
Do you have any questions?
'The only thing that is constant is change.’

~ Albert Einstein ~

CREDITS: This presentation template was created by Slidesgo, and includes icons by Flaticon, and infographics & images by Freepik
64