Email Address Internationalization (EAI)
Configuration and Technical Overviewहरीश चौधरी (Harish Chowdhary), UA Ambassador
मेल@ईंटरनेट.भारत, Intern3tgovernance@gmail.com
Universal Acceptance2Key Fundamental Aspects : Unicode, IDNs, UA
Encoding glyphs into code points.
There are multiple ways to use a glyph“è” = U+00E8“e`” = “è” = U+0065 U+0300
Normalization is a process to insure that no matter the user type, the end
representation will be the same.
For the two entries above, Normalization Form C (NFC) will generate
U+00E8for both▪ In specifications, code points are shown in hex using the U+XXXX notation.
▪ Code points are typically carried using the UTF-8 (Unicode Transformation
Format, 8 bit) format.
a. Variable number of bytes for a single code point.
b. ASCII is used as is.
c. Gold standard for carrying Unicode code points in web, protocols, etc▪
▪

3

Domain Names▪
▪
▪
▪

A domain name is an ordered set of labels: a.b.c.dA top-level domain is the rightmost label: “d” in left-to-right scripts.
The Domain Name System (DNS) is the distributed database and service forquerying domain name records.
A domain name may have multiple DNS records such as:
IPv4 address for that domain name.
IPv6 address for that domain name.
Hostname of the email server responsible for that domain name and so on….
▪ A zone is the list of domain name records – called Resource Records (RR) – forthe labels under another label (a bit simplified…)

4

Types of Domain Names
Domain Name

Category

universal-acceptancetest.international

ASCII.ASCII, new-longuniversal-acceptance-test.icu

ASCII.ASCII, new-short‫تجربة‬-‫القبول‬-‫الشامل‬.‫موريتانيا‬सार्वभौमिक-स्र्ीकृति-परीक्षण.संगठन
ཡོངས་ཁྱབ་ངོས་ལེན་བརྟག་དཔྱད.com

IDN.IDN, RTL
IDN.IDN, Devanagari

universal-acceptance-test.‫קום‬ASCII.IDN, RTL

Email Address

Category

email-test@universalacceptance-test.international

ASCII@ASCII.ASCII, new-longਈਮੇਲ-ਪਰਖ@ਸਰਵਵਵਆਪਕ-ਪਰਵਾਨਗੀਪਰਖ.ਭਾਰਤ

Unicode@IDN.IDN, , Gurmukhi

email-test@universalacceptance-test.‫קום‬ASCII@ASCII.IDN, RTL, Hebrew

IDN.ASCII, Tibetan

5Internationalized Domain Names (IDNs)▪
▪
▪

Internationalized Domain Names (IDNs) enable the use of non-ASCII
characters for any label of a domain name.
Not all labels of a domain name may be internationalized.
Example: exâmple.ca
User uses the IDN version, but the IDN is converted into ASCII for DNS
resolution.
exâmple => exmple-xta => xn--exmple-xta
The xn-- prefix is added to identify an IDN

Example process of using an IDN
▪
▪
▪

▪
▪

User enters in a browser: http://exâmple.ca
Browser does normalization on the user entry.
Browser converts exâmple.ca in an ASCII compatible representation called
Punycode [RFC3492], and adds ‘xn--’ in front of it
exâmple.ca becomes: xn--exmple-xta.ca or तनक्सी.भारि becomes:
xn--11b1b9b0ah0f.xn--h2brj9c
Browser calls the DNS to get the IP address of xn-- exmple-xta.ca
Browser connects to the HTTP server at the received IP address

6

Internationalized Domain Names▪The protocol for handling IDNs is named IDN for Applications (IDNA)▪Two versions: IDNA2003 and IDNA2008. The latter (IDNA2008) is the one currently
used.
▪U-label is the Unicode native representation of an IDN label: ननक्सी.भारत
▪A-label is the Punycode representation of an IDN label: xn--11b1b9b0ah0f.xn-h2brj9c

7

Universal Acceptance (UA)UA is about how to appropriately support internationalized identifiers, as well as
new and long TLDs.
Internationalized identifiers
* IDN
* EAI

8

How to Quantify Universal AcceptanceAcceptValidate

Process

Store

Display

Applicable to both domain names and email addresses

9• Should accept both
A-label and Ulabel.
• Support UTF-8• Need large enoughinput field withtotal 735 bytes
requirements
including email
local part of up to
64 bytes• Input to benormalized as perUnicodeNormalization
Form C (NFC)• Validation against
the IDNA 2008
protocol• TLD verification
against theauthoritative
maintained by
IANA.
• Local part of email
is defined byservice providers,
however comply tobest practices• All components ofdomain name oremail(except the
TLD name if it is
not an IDN) should
be in a single script(e.g., Arabic orHan) or closely
related scripts(e.g., Japanese
Kanji, Katakana,Hiragana, andRomaji).

• Use NFC form
before storing.
• Use UTF-8 forstorage in
databases.
• Ensure
environment
supports Unicodeand automaticconversion to orfrom UTF-8 on
input and output.
• Consistent internal
representation—either U-labels or
A-labels—for
IDNs.
• Use most current
Unicode standards.
• Regardless of theway addresses anddomain names are
stored, you must
be able to match
strings in multiple
formats.

• Use recent
versions of s/w
libraries and web
standards• Use APIs that
support UTF-8• RTL scripts
require special
considerations• Allow as many
scripts that theUnicode supports.
• Be aware thatsome languages
can be written
using different
scripts, and thatsome scripts canbe used to write
many different
languages.
• Ensure sort order,
searches andcollation according
to locale andlanguages
specifications• Ensure that s/w,
applications
displays nearly all
Unicode code
points, with
relevant fontlibraries.
• Display IDNs intheir native
character form
unless there is a
specificrequirement todisplay them as Alabels.
• Specialconsideration
while using LTR
and RTL scripts..
• It may necessitate
to design
applications
separately for
different languages
or language
groups.
10

What Comes Under the UA Ambit?
Applications and Websites- Wikipedia.org, ICANN.org, Amazon.com, custom websites globally- PowerPoint, Google Docs, Safari, Acrobat, custom apps
Social Media and Search Engines- Chrome, Bing, Safari, Firefox, local (e.g., Chinese) browsers- Facebook, Instagram, Twitter, Skype, WeChat, WhatsApp, Viber
Programming Languages and Frameworks- JavaScript, Java, Swift, C#, PHP, Python- Angular, Spring, .NET core, J2EE, WordPress, SAP, Oracle
Platforms, Operating Systems and Sytem Tools- iOS, Windows, Linux, Android, App Stores- Active Directory, OpenLDAP, OpenSSL, Ping, Telnet
Standards and Best Practices- IETF RFCs, W3C HTML, Unicode CLDR, WHATWG- Industry-based standards (health, aviation, ...)

11

Anatomy of an IDN andInternationalized Email
Internationalized Domain
Name (IDN)Internationalized Email

http://www.bücher.berlin

Günte@bücher.berlin

ProtocolSubdomain

DomainTLD

Local part

Domain name12

Actions to be Performed on a Domain Name
List of actions needed to handle IDNs effectively:• Normalization ( é U+0065 U+0301 → é U+00E9)• U-Label~A-Label Conversion ( ‫الہور‬to xn--mgbu9cr81d)• Length validation of A-Label• Case-folding (if necessary) (Ωω ,Éé )• Validation (ककिाबकि िाब )• Unicode Compatible Display (ဟယ်လို �������

)13

Basics of Unicode and its Forms•

Unicode encodes different scripts in a machine understandable
manner i.e., in numerical form called a “Codepoint.”• These Codepoints are in hex form and use the U+XXXX notation,
the XXX being the assigned code point value.
Though Unicode aims at adhering to the one character, one codepoint principle, there have been instances where it has ended upproviding multiple ways of encoding single code point.

•è = U+00E8
e + ` = è = U+0065 +U+0300•क़ = U+0958
क + ़ = क़ = U+0915 + U+093C‫ =آ‬U+0622‫ا‬+ ٓ = ‫ =ا‬U+0627 +U+0653

Normalization ensures that the end representation is the sameeven if users type differently.
• IDN standards recommend using Normalization Form C (NFC).
14

Email Terminology
UASG Working Groups

Role

Mail User Agent (MUA)

The software used by the user who sends andreceives email.
With web mail, the MUA is an application run in abrowser environment. E.g. Gmail or Xgen Running in
Browser/outlook/thunderbird

Mail Transfer Agent (MTA)

A server application that receives mail from the MSA, or fromanother MTA. It will find (through name servers and the DNS) the
MX record from the recipient domain's DNS zone in order to know
how to transfer the mail. It then transfers the mail (with SMTP) toanother MTA (which is known as SMTP relaying) .
Examples of MTAs are Postfix, Exim, Sendmail, qmail etc.

Mail Submission Agent (MSA)

A server program that receives mail from an MUA, checks for any
errors, and transfers it (with SMTP) to the MTA hosted on the same
server. Typically, this function is bundled with an MTA

Mail Delivery Agent (MDA)

A software, usually on servers, which receives theemail from an MTA and is the final destination for
the email.
It typically stores the email in a file (or a database) and waits for the
MUA of the destination user to fetch the email. Typically, thisfunction is bundled with an MTA.
An example is Dovecot, which is mainly a POP3 and IMAP server
allowing an MUA to retrieve mail, but also includes an MDA whichtakes mail from an MTA and delivers it to the server’s mailbox
15

Internationalized Email Acceptance
Rate

Testing of popular websites taken from UASG027
16

Email Terminology
For example, Postfix is typically used as an MTA, MDA, and MSA.
Sending an Email17

Email : How To Find the Destination Server
When sending email to user@example.com, the method to find the destination
email server is by querying the DNS for the MX records of the domain.
For example, the MX records for example.com could be:• MX 10 server1.example.com• MX 10 server2.example.com• MX 20 server3.example.com
The sender email server would then try connecting to either server 1 or server 2
since they have same priority (10)

If none respond, it would then try server 3 since it has a lower priority (20)The higher number means lower priority

18

Email Address Internationalization (EAI)
Email syntax: leftside@domainname▪Domain name can be internationalized as an IDN (U labels or A-labels).
▪Left side (also known as local part/mailbox name) with Unicode (UTF-8) is EAI.
▪Not all email servers support EAI, so a negotiation protocol is used to only send
EAI when the target server supports it. If not, then it falls back and returns an‘unable to deliver’ message back to the sender.
▪The SMTPUTF8 option is used within the mail transfer protocol (SMTP: Simple
Mail Transport Protocol) to signal the support of EAI by an email server
Overhead▪Mail headers need to be updated to support EAI.
▪Mail headers are used by mail software to get more information on how todeliver email.

19

EAI Protocol Changes
SMTP▪Is augmented to support EAI.
▪Has a signalling flag (SMTPUTF8) to specify support of EAI.
▪All SMTP servers in the path must support EAI to successfully deliver the email.
POP/IMAP▪Are augmented to properly support EAI.
▪Have a signalling flag to specify support of EAI.
Could “half support” EAI by providing a downgraded email version to the non-EAI
conforming email software clients (downgrading - While it may look interesting,downgrading may cause many issues for the users and the sysadmin to debug issues. Try to avoid
using the downgrading mechanism if you can. ).

EAI Protocol Changes : SMTP
▪SMTP Server announcing the support of EAI on the initial greeting.
EHLO SMTPUTF8▪SMTP Client connecting to the compliant SMTP Server.
MAIL SMTPUTF8▪Headers may have UTF-8 content.
▪Email body already supports UTF-8.
EAI IMAP/POP Protocol Changes
POP : UTF8 command
IMAP : ENABLE UTF8=ACCEPT command20

Protocol Changes, Delivery Path Considerations

To send and receive an email with EAI▪All email parties involved in the delivery path have to be updated for EAI
support.
▪If a single SMTP server in the path does not support EAI, then the email is notdelivered.

21

Demonstration : Setup

EAI email path between two (2) users.
User 1:kévin@exâmple.com▪Using MailMate on MacOSX with SMTPUTF8 enabled.
▪Using his own mail server infrastructure.
User 2:
peter@exâmple.ca▪Using Gmail web interface: no configuration necessary apart from thedomain and user.
▪Using Google mail server infrastructure.

22

Demonstration : Setup
For User 1 (kévin@exâmple.com)▪Its inbound email server is Courier as IMAP server.
▪Its outbound email server is Courier as SMTP server.
▪Courier uses a Postfix relay server acting as a pure MTA.
For User 2 (peter@exâmple.ca)▪Its inbound/outbound email servers are Google Gmail servers.
Confirming advertised mail server (Postfix) for kévin@exâmple.com:
dig xn--exmple-xta.com mx
xn--exmple-xta.com. 300 IN MX 10
postfix.xn--exmple-xta.com.
Confirming advertised mail server (Gmail) for peter@exâmple.ca:
dig xn--exmple-xta.ca mx

23

EAI : Configuration
Postfix: Configuration
These are the specific EAI configuration requirements.
/etc/postfix/main.cf...
# enable SMTPUTF8
smtputf8_enable = yes#defines the hostname for SMTP. Does not need to be an IDN.
myhostname = postfix.xn--exmple-xta.com#defines the domain of the host. Does not need to be an IDN.
mydomain = xn--exmple-xta.com#domains of the user mailboxes. With smtputf8_enable, this is the key config.
virtual_mailbox_domains = exâmple.com

24

EAI : Configuration
Courier: Configuration
These are the specific EAI configuration requirements.
/etc/courier/defaultdomain# defines the default domain used by Courier
exâmple.com/etc/courier/locals# defines the local domain Courier will deliver mail toexâmple.com
localhost25

EAI : Configuration
MailMate: Configuration
Commercial email client on
MacOSX.
❑For user kévin@exâmple.com❑Usingcourier.xn--exmplexta.com running Courier software
as its IMAP and SMTP server.
❑Set support for SMTPUTF8:▪Defaults
write
com.freron.MailMate
MmSMTPUTF8Enabled –bool
YES

26

From MailMate to Gmail

27

Important points
Case Folding
Mail sent from peter@viagenie.ca to KÉVIN@exâmple.com•
Email was NOT delivered.
Why?
kevin and KEVIN as local parts are typically handled as same user. Case folding is
automatic for ASCII local parts.
But kévin and KÉVIN are by default not the same user. Unicode case folding istypically not done by servers.

Typically in ASCII email addresses, the local part can be uppercase or lowercase.
In UTF-8, case folding is not the same as ASCII.

28

Important points
Avoiding Interpretation as Spam
▪Spam filtering may think EAI is spam even with SPF, DKIM, etc.
•This is because the spam filtering does not know about EAI.
▪New TLDs and IDNs may also be viewed as spam.
▪Spam filtering may think EAI is spam even with SPF, DKIM, etc. This is because
the spam filtering does not know about EAI.
▪New TLDs and IDNs may also be viewed as spam. Some Mail User Agent (MUA)
software uses your contacts database to determine if an email is known to you
or not▪This will make it more likely to be considered spam if the email of the sender is
not in your contacts database.
▪Make sure the contacts database contains the EAI email of the contacts.

29

Enabling UA the Python
Way
Coding for UA compliant domain
and email validators

About Python Programming Language•••

Python is an open-source language operating under OSIapproved open-source license.
A general purpose, interpreted, high level programming
language with a very vibrant developer support community.
Most of the code/packages are open- source, at least all thoseconcerning UA.

31

Python 2: Datatypes and Unicode Enablement•

The prevalent Python version is Python 3, however, there aremany older implementations which still use Python 2 due tolegacy reasons.
• As per the survey*, Python 2 is still employed by 6% of Python
implementations.
• In Python 2, the default encoding for the String Type “str” wasASCII.
• The Unicode encoding required to be explicitly declared by
prepending “u” before the string literal.
• It used to show datatype of such a string as “unicode”, insteadof the default “str”• Python had a major Unicode support enablement in Python 3• *Python Developers Survey 2020 conducted by JetBrains

ACCEPT

VALIDATE

STORE

PROCESSDISPLAY

32

Python 3: Datatypes and Unicode Enablement
Python 3 by default uses UTF-8 for its entire source code.
The primary string datatype “str” thus is by default Unicode
enabled.
• In Python 3 onwards, one can even use Unicode in the string
literal, meaning, one can have Unicode variable names too.
• For more information, refer to the Python official
documentation on Unicode support here.
• In short, the basic “str” datatype in Python is sufficient tocater to Unicode handling requirements.
••

ACCEPT

VALIDATE

STORE

PROCESSDISPLAY

33

Unicode Normalization••

Though Unicode aims at adhering to the one character, one codepoint principle, there have been instances where it has ended upproviding multiple ways of encoding single code point.
è = U+00E8

क + ़ = क़ = U+0915 + U+093C

e + ` = è = U+0065 +U+0300क़ = U+0958‫ = آ‬U+0622‫ا‬=U+0627 + U+0653

Normalization ensures that the end representation is the same,even if users type differently.
• IDN standards recommend using Normalization Form C (NFC).
ACCEPT

VALIDATE

STORE

PROCESSDISPLAY

34

IDNA Conversion•

The default python library “encodings.idna” for IDN
Compatability Processing, is based of IDNA2003
implementation. It is strongly recommended not to use thelibrary.
•

Ref: https://docs.python.org/2.4/lib/module-encodings.idna.html•

Alternatively, the “idna” package present on Python Package
Index (PyPI) ably supports the IDNA2008 implementation.

•

Can be installed in two ways:• pip install idna
• Downloading files from https://github.com/kjd/idna
followed by:• "python setup.py install"•

The “idna” library is highly recommended by the UASG for UA
compliant code development by developers.
35

IDNA Conversion – IDNA Package• There are two major functions:• Encode: converts the IDN to its Punycode equivalent• Decode: converts the Punycode domain to its IDN equivalent• There are three modes of operation:• idna.encode(<IDN_STRING>) & idna.decode(<PUNYCODE_STRING>)• E.g., idna.encode(exâmple.ca’) or idna.decode('xn--exmple-xta.ca')• <IDN_STRING>.encode('idna’) & <PUNYCODE_STRING>.decode('idna')• Here variables holding <IDN_STRING> and <PUNYCODE_STRING> will be
used• idna.alabel(‘UNICODE_Label') & idna.ulabel(‘PUNYCODE_LABEL’)• idna.alabel('périple') & idna.ulabel('xn--priple-bva')• For bug-report/issue-tracking, please visit:https://github.com/kjd/idna/issues
ACCEPT

VALIDATE

STORE

PROCESSDISPLAY

36

Storing as U-Label or A-Label• As long as the labels are normalized, the U-Label and
A-Label are fully interconvertible and can be stored on
the database in either form.
• However, if the business requirement dictates some
operations which depend on various forms of Unicode
processing (Validating, sorting, searching), it becomesincumbent on the developer to store in U-Labelformat.
ACCEPT

VALIDATE

STORE

PROCESSDISPLAY

37

Displaying the Unicode Properly• Make sure that your applications/interfaces support thenecessary text-layout engines viz. Uniscribe / DirectWrite(Windows), Pango and ICU (Linux), AAT (IOS). Most of theplatforms/languages already support them implicitly. This is
more relevant for legacy applications.
• Always ensure U-Label for user-facing interfaces unless ALabel is explicitly needed.
• Make sure you use appropriate fonts for ensuring proper
rendering and legibility for the end user.

ACCEPT

VALIDATE

STORE

PROCESSDISPLAY

38

Displaying Scripts from Right-to-Left (RTL)

ACCEPT

VALIDATE

STORE

PROCESSDISPLAY

39

Validators
Domain and Email Validation
ACCEPT

VALIDATE

STORE

PROCESSDISPLAYDomain Name Validation•

Domain name validation typically involves checking if theuser-submitted domain name complies with the URL protocol(protocol compliance validation check).

•

The user-submitted domain resolves to a registered domainname (functional validation check).

ACCEPT

VALIDATE

STORE

PROCESSDISPLAY41

Domain Validation•

Validators Package• Python Data Validation for Humans•

Domain validation is RegEx-based• With very rudimentary support for Punycode inclusion.

•

Email Validation is also RegEx-based• Seems to give results but other important tasks likenormalization and cannot be expected.

•

Better to be avoided until the required features are well
supported.

42

Domain Validation•“idna” package available through PyPI•

Provides the “validation” through exception mechanism(typically not the expected way though).
>>> import idna>>> idna.encode('Königsgäßchen')•Raises exception:••

idna.core.InvalidCodepoint: Codepoint U+004B at position 1 of'Königsgäßchen' not allowed

Likewise, the library raises various exceptions as per the
specifications from the IDNA2008 specification.

43

Domain Validation••“idna” package available through PyPI

More specific exceptions that are generated are:• idna.IDNABidiError - when the error reflects an illegal
combination of left-to-right and right-to-left characters ina label.
• idna.InvalidCodepoint - when a specific codepoint is an
illegal character in an IDN label (i.e.. INVALID).
• idna.InvalidCodepointContext - when the codepoint isillegal based on its positional context (i.e., it is CONTEXTOor CONTEXTJ but the contextual requirements are notsatisfied).

44

Domain Validation•

idna.IDNABidiError - when the error reflects an illegal
combination of left-to-right and right-to-left characters in a
label
•
•

•

>> import idna>> idna.encode('कख‫ب‬.com')Raises exception:• IDNABidiError('Invalid direction for codepoint at position{0} in a left-to-right label'.format(idx))• idna.core.IDNABidiError: Invalid direction for codepoint
at position 3 in a left-to-right label

45

Domain Validation

idna.InvalidCodepointContext - when the codepoint is illegal
based on its positional context (i.e., it is CONTEXTO or
CONTEXTJ but the contextual requirements are not satisfied).

•••>> import idna>> idna.encode('wordstar.com')•••

CONTAINS ZERO WIDTHJOINER, A CONTEXTJ Characterw o r d ZWJ s t a r . c o mRaises exception:••raise IDNAError('Unknown codepoint adjacent to joiner {0} at
position {1} in {2}'.format(
idna.core.IDNAError: Unknown codepoint adjacent to joiner
U+200D at position 5 in 'word\u200dstar'46

TLD Validation
Validating the TLD part with IANA repositoryं ट्रान्स.भारत
https://थ क
https://<DOMAIN_NAME>.<TOP_LEVEL_DOMAIN>•

Validate the TLD part of thedomain name directly with the
IANA repository.

•

TLD allocation process involves
many other processes thanmere IDNA protocol compliance.

•

Checking directly with the IANA
repository will always yield themost accurate results on the
date.
47

TLD Validation
Validating the TLD part with IANA repositoryं ट्रान्स.भारत
अक्षि@थ क<LOCAL_PART>@<DOMAIN_NAME>.<TOP_LEVEL_DOMAIN>
SAMPLE CODE FOR VALIDATING ASCII/A-LABEL TLDs WITH LIVE IANAREPOSITORY:For IDN TLDs, always
import urllib3
convert them to their
Punycode equivalent,
http = urllib3.PoolManager()
uppercase, before
strTLD = "XN--H2BRJ9C"
checking
resp = http.request("GET", "https://data.iana.org/TLD/tlds-alpha-bydomain.txt")
strValidTLDs = resp.data.decode().split('\n')if(strTLD.upper() in strValidTLDs):
print ("TLD FOUND!")
else:
print ("TLD NOT FOUND!")

Take appropriate
action here as per yourcode logic

SAMPLEINPUTOUTPUTAMAZONTLD FOUND!

XN--H2BRJ9C

TLD FOUND!

ABCDETLD NOTFOUND!

CAUTION: CONVERT U-LABEL TO
A-LABEL BEFORE CALLING THIS!

48

Python 3: Socket, Hostname Resolution
socket.gethostbyname(hostname)
Does not support IPV6
Addresses

socket.getaddrinfo(host, port, family=0, type=0, proto=0, flags=0)
Supports IPV4/IPV6 dual stack

Both above methods take A-label as well as the U-label
as the host name.
• Raises “exception socket.gaierror” if provided Punycode
instead of the U-label.
•

49

Python 3: Socket, IP Resolution
socket.getfqdn([name])

Returns a fully qualified domain namefor name. If name is omitted or empty, it isinterpreted as the local host.

socket.gethostbyaddr(ip_address)

Returns a triple (hostname, aliaslist,
ipaddrlist). Supports both IPV4 and IPV6
Addresses• Both above methods take the U-label as the host name.
• Raises “exception socket.gaierror” if provided Punycode
instead of the U-label.

50

Major UA Challenges•

Some applications are still verifying domain names
incorrectly by using one of the outdated methods:• Check for a fixed length of TLD between 2-4 characters (TLD
can be up to 63 characters).
• Check from a fixed set of TLDs, e.g., using static list of
strings.
• Check for only ASCII characters.

•

Some applications do not cater to additional requirementsfor validating EAI:
• Check mailbox name to be a valid string in UTF-8 format.
• DomainName can be ASCII or IDN.

51

UA-Readiness Testing Framework

AT: Accept test

ST: Store test

VT: Validate test

P2T: Process test on the output

P1T: Process test on the input

DT: Display test

Detailed UA-Readiness Testing Framework is available in the form of aTechnical Guide UASG026 here.

52

Validating User Input
Validating user input, or any input, is extremely useful forvarious reasons, some of which include: a better user
experience, increased security, and avoiding irrelevant issues.
• Validating domain names and email addresses is useful.
• Some validation methods for domain names and email
addresses:• Basic syntax checks: is the syntax of the string correct?
• Does the domain name contain ‘.’ ?
• Does the email address contain ‘@’ and a valid domainname part?
• Functional checks: does the domain name or email addresswork?
• Is the top-level domain (TLD) in use?
• Is the whole domain name in use?
• Is the email in use?
•53

Validating Domain Names
Validating syntax:• ASCII: RFC1035• Composed of letters,
digits, and hyphen.
• Max length is 255 octets
with each label up to 63
octets.
• IDN: IDNA2008 (RFCs 58905894)• Valid A-labels• Valid U-labels

Validating function:• Is the TLD in use?
• Verify against the list of
TLDs.
• Verify using a DNS
request.
• Is the whole domain name inuse?
• Verify using a DNS
request.

54

Validating Email Addresses
Validating syntax:• An email address iscomposed of:
mailboxName@domainName• Validating syntax:• For domainName, seeearlier discussion.
• For mailboxName:• ASCII• UTF8 (for EAI)

Validating function:• Is the domain name set
up to send and receive
emails?
• Is the mailbox name able
to send and receive
emails?

55

Email Validation

Composition of an Email<LOCAL PART> @ <DOMAIN NAME>••

Has to be validated as per thegeneral guidelines laid down in
the RFCs.
A good guide to the same isreleased by the UASG titled“UASG 028 Considerations for
Naming Internationalized Email
Mailboxes EN” . It can beaccessed here.

• Has to be validated as per thegeneral “Domain Name
Validation” process, as
discussed.

57

Email Validation
Validating an Email is much more than just validating for certain
characters.
<LOCAL_PART>@<DOMAIN_NAME>.<TOP_LEVEL_DOMAIN>

Each of the above get governed by different sets of Internet
standards/processes.
LOCAL_PART: Gets governed by the policies of the Email Service
Providing Entity, typically the domain owning body.
Please refer to best practices recommended by the
UASG here.
DOMAIN_NAME: Gets governed by IETF Protocols (IDNA 2008) for
domain names.

TOP_LEVEL_DOMAIN: IANA TLD Repository

58

Email Validation•

A python developer should avoid the temptation of using an“re” package and begin the first attempts at validating an email
address with naïve RegEx’s like below:'^[a-z0-9]+[\._]?[a-z0-9]+[@]\w+[.]\w{2,3}$’• For obvious reasons, in the context of IDNs, validating for only“a-z” is a non-starter.

• So is validating domain name and top-level domain (TLD) witha “\w”.

59

Email Validation•••

Since individual parts of the email ID is governed by different
entities with different policies, it is best to delegate thevalidation function of an email ID to a proper dedicatedlibrary.
The Python community is rich with options when it comes tousing open-source libraries which “just work”.
Here are some of the options at hand:• email-validator 1.0.5• pylsEmail 1.3.2

60

Email Validation
email-validator: (Current version 1.1.3)• Basic syntax checking• Checks resolvability• Supports IDN email by including SMTPUTF8 in resolvability
check and Internationalized Local part check• For storing email addresses, provides “Normalization”
option• Complies with IDNA2008
Installation: pip install email-validator

61

Email Validation
email-validator: (Current version 1.1.3)• Checks if the domain part of the email resolves to a valid
domain name.
>>from email_validator import validate_email, EmailNotValidError>>valid = validate_email("user@mydomain.tld")>>print(valid.email)The domain name mydomain.tld does not exist.
>>valid = validate_email("propriétaire@rhône.hôtel")>>print(valid.email)propriétaire@rhône.hôtel

62

Email Validation
email-validator: (Current version 1.1.3)• Provides normalized form of the email part• This example uses a non-NFC character• a + ̂ (U+0061 + U+0302) (Normalizes to â – U+00E2 )>>from email_validator import validate_email, EmailNotValidError>>valid = validate_email("utilizator@exâmple.ca")>>print(valid.email)>>print(email)Non-Normalized
String"utilizator@exâmple.ca"
Normalized
String63

Email Validation
pylsEmail : (Current version 1.3.2)• This package is supposed to be doing purely protocol
checking for an email.
• However, it does not seem to support EAI at the moment.
>>from pyisemail import is_email>>address = "utilizator@exâmple.ca">>bool_result = is_email(address)>>detailed_result = is_email(address, diagnose=True)>>print(bool_result)>>print(detailed_result)
False<InvalidDiagnosis: EXPECTING_ATEXT>

64

Test Cases for IDNs and Internationalized
Email Addresses
Available in: “UASG 004 Use Cases for UA Readiness Evaluation EN”
Link: https://uasg.tech/download/uasg-004-use-cases-for-uareadiness-evaluation-en/

65

Mail Software and Services
EAI Support▪MailMate (MUA) on MacOSX: v.1.9.4 minimum▪Postfix (SMTP) v3.0 minimum•http://www.postfix.org/SMTPUTF8_README.html▪Courier (IMAP, POP, SMTP) v1.0 minimum (IMAP version 5.0.8)
EAI NOT supported (as of October 2019):▪Dovecot (IMAP, POP)
▪Zimbra▪Mozilla Thunderbird
Services supporting EAI▪Gmail▪OutlookMore info in UASG021A available at www.uasg.tech

66

Conclusion❑EAI is essentially supporting UTF-8 local parts of an email address.
▪Which also means supporting in the headers.
▪Requires changes to mail servers and mail clients.
❑All SMTP servers in the path must be EAI-enabled to deliver the mail to thedestination.
❑Courier and Postfix are two open-source software that support EAI with very
limited changes in configuration.
❑In configuration files, be careful to use A-labels, U-labels, or both, dependingon the situation.

67

Additional Information for EAI, IDNs, UA❑Universal Acceptance Steering Group (UASG)
https://uasg.tech
https://www.icann.org/ua❑Internationalized Domain Names (IDNs)
http://icann.org/idn❑UASG028​ : Consider​ations for Naming In​ternationalized Emai​l Mailboxes•The document is intended for email systems administrators to help themprovision mailboxes, configure, and manage systems compatible withinternationalized email addresses.
•It outlines the considerations for naming internationalized mailboxes andhelps administrators make good choices when setting their mailbox
names policy.

68

69

70