Project: talenddataquality
Issue TDQ-9788: Email validation component - Step 1
What is an email: http://en.wikipedia.org/wiki/Email_address Create an API and a component called *tVerifyEmail* under Data_Quality By default, this component will check a generic email regex: {noformat}"^\s*(([^\s\p{Cntrl}\(\)<>@,;:'\\\"\.\[\]]|')+|("[^"]*"))(\.(([^\s\p{Cntrl}\(\)<>@,;:'\\\"\.\[\]]|')+|("[^"]*")))*$" {noformat} In the advanced setting, this regex can be edited. Parameters in the component will help the user to specify further the email format. The user will be able to specify the local part (string before the '@' sign). Either by writting a simplified expression such as - w that means there is a word - a.a that means "letter\.letter" - a.w that means "letter\.letter+" - 1.w that means a digit followed by dot followed by a word. - w1.w means word followed by one digit followed by dot followed by a word - wN means word followed by a numeric - w[-_]w means word followed by either - or _ then word - a* means any number of characters after a letter. An option (checkbox) enables the user to decide whether the local part can be case sensitive or not (the domain part is never case sensitive). When this option is checked, then we must distinguish between "Aw" which means one word starting with an upper case and "aw" which is equivalent to "w" and means a word in lower case. the user must be able to define a list of valid domains or a list of invalid domains (he can't define both at the same time). The domain is the string after the '@' sign. A black list is the list of all domains that are not valid: every email with a domain from the blacklist must be invalid In the case of blacklist, when the TLD is invalid, then no need to check other conditions, the email is invalid A white list is the list of all acceptable domains: every email with a domain NOT in the list is invalid In the case of the whitelist, there is no need to check the TLD (it's up to the user to provide the correct domain part of the email) Optionally, this component will reuse code from the email UDI to check the email via socket queries. Think about the reusability of the API in some Java DQ indicator (later). _(it has been suggested to reuse the same syntax as the one used in tStandardizeRow, but it's more readable to have something like aw than LETTER WORD and it's clear that there is no space in aw whereas it's not so clear in "LETTER WORD". Therefore, I suggest we keep the above syntax or a similar one. )_ In this first step, the output schema will have only one fixed columns: "VERIFICATION" The content of the VERIFICATION column can be: - verified (meaning that the email complies to the given rules) - invalid (meaning that the email does not comply to the rules) See some existing code at http://leshazlewood.com/2006/11/06/emailaddress-java-class/ http://commons.apache.org/proper/commons-validator/apidocs/org/apache/commons/validator/routines/EmailValidator.html These codes may have performance issues or other problems that the TLD list is not complete. We should provide an option for the user to choose whether the algorithm must validate TLD names or not. Or maybe, we need to allow the user to add his own valid TLD names. See https://data.iana.org/TLD/tlds-alpha-by-domain.txt
Story points: 40