Repository: datactive/bigbang (language: python)

Issue:
identify institutional affiliation of mail senders
For a mail sender, have an automated way of identifying their institutional affiliation.

Possible sources of information: 
- the email domain
- email signatures
- LinkedIn or other external data lookup

Affiliation data format and parsers
- Develop a portable data format for storing and transferring information about the affiliations between individuals and organizations
- This will be a special case of a more general serializable data format that is both machine- and human- readable (e.g. YAML)
- Deliver this format, and tooling for parsing and writing it in Python, as code in BigBang

Affiliation data format and parsers
- Develop a portable data format for storing and transferring information about the affiliations between individuals and organizations
- This will be a special case of a more general serializable data format that is both machine- and human- readable (e.g. YAML)
- Deliver this format, and tooling for parsing and writing it in Python, as code in BigBang

identify institutional affiliation of mail senders
For a mail sender, have an automated way of identifying their institutional affiliation.

Possible sources of information: 
- the email domain
- email signatures
- LinkedIn or other external data lookup

identify institutional affiliation of mail senders
For a mail sender, have an automated way of identifying their institutional affiliation.

Possible sources of information: 
- the email domain
- email signatures
- LinkedIn or other external data lookup

identify institutional affiliation of mail senders
For a mail sender, have an automated way of identifying their institutional affiliation.

Possible sources of information: 
- the email domain
- email signatures
- LinkedIn or other external data lookup

identify institutional affiliation of mail senders
For a mail sender, have an automated way of identifying their institutional affiliation.

Possible sources of information: 
- the email domain
- email signatures
- LinkedIn or other external data lookup



Discussion / hints:
See #88 for signature detection issue

see also #20 
Now we have `ietfdata` as an alternative data source for affiliations. This can be used for training data for machine learning or other entity resolution techniques.
These are the top 50 most frequent email domains for the senders to the working group `httpbisa`:

gmail.com              6530
mnot.net               4117
gmx.de                 3606
phk.fre                1193
treenet.co             1006
qbik.com                843
henriknordstrom.net     790
google.com              732
chromium.org            700
gbiv.com                697
ietf.org                629
microsoft.com           614
intalio.com             420
apple.com               391
shareable.org           375
opera.com               366
greenbytes.de           329
mozilla.com             329
adambarth.com           313
ducksong.com            305
gmx.net                 282
kerwin.net              266
briansmith.org          256
laposte.net             256
xpasc.com               255
cisco.com               246
w3.org                  243
belshe.com              237
redhat.com              206
iaea.org                203
lukasa.co               203
cs.tcd                  202
it.aoy                  202
twitter.com             190
haxx.se                 183
ericsson.com            161
crf.can                 159
xyzzy.cla               150
cryptonector.com        146
zinks.de                134
elisanet.fi             127
rtfm.com                122
isode.com               120
computer.org            115
osafoundation.org       109
acm.org                 108
bisonsystems.net        104
bbc.co                   99
textuality.com           99
nygren.org               97
Obviously IETF is an uncommon space compared to many in that several of these domain names are individuals I recognize, rather than companies or email services. (It's notable that Mark, one of the chairs, has almost as many emails as all gmail users combined!) 

That some prominent individuals use their own domains in their email addresses also suggests that that limitation can be addressed by accessing their affiliation through other sources (affiliation listed in authored documents, or through a specific profile in the IETF datatracker).
I wonder if we can actually leverage the semantics of TLDs --  .com, .org, .net -- for our analysis.

It's a good point that there's a 1-to-many relationship between domains and people, but that it's a very uneven mapping. That coudl be a useful cue for whether a domain is for an organization, vs. a generic email host or a single person.

Extracting affiliation <-> email mappings from the datatracker would certainly help with this.
The only issue is that it is a benefit that's restricted to the IETF domain.
It could be used for training some other classifier though.
@npdoty @nllz I wonder if you could give me your gut check on this.

I've been working on a metric to guess whether a domain name corresponds to a significant organization vs. an individual.

When you eyeball this plot, do you see an intuitive difference between the domain names at the top of the plot vs. at the bottom of the plot?

![image](https://user-images.githubusercontent.com/68752/109816144-67e67c00-7bfe-11eb-9725-aca78f1c26b5.png)

What is that intuitive difference?
Hi Seb,

Thanks for this. The way I interpret this graph is the amount of different people that email from that specific domain. 

For example: anviwalrusden.com is the personal domain of Andrew Sullivan: former IAB chair and very active long time IETF participant. Google, Oracle, Microsoft, Nokia, etc on the other end of the spectrum have a lot of different people working in IETF, and thus many different email address contributing to discussions from that domain. 

Hope this helps.

Best,

Niels 

On 03-03-2021 14:57, Sebastian Benthall wrote:
> @npdoty <https://github.com/npdoty> @nllz <https://github.com/nllz> I wonder if you could give me your gut check on this.
> 
> I've been working on a metric to guess whether a domain name corresponds to a significant organization vs. an individual.
> 
> When you eyeball this plot, do you see an intuitive difference between the domain names at the top of the plot vs. at the bottom of the plot?
> 
> image <https://user-images.githubusercontent.com/68752/109816144-67e67c00-7bfe-11eb-9725-aca78f1c26b5.png>
> 
> What is that intuitive difference?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub <https://github.com/datactive/bigbang/issues/25#issuecomment-789731775>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCWQ5Z7W647NDMTQRV7ZWTTBY537ANCNFSM4ARFTWQA>.
> 

-- 
Twitter         :   @nielstenoever
PGP fingerprint :   8D9F C567 BEE4 A431 56C4 
                    678B 08B5 A0F2 636D 68E9

Thank you. Yes, that's what I was hoping it would get at.

For these people like Andrew Sullivan and Mark Nottingham, do they use a personal domain to signify that they are participating in their personal capacity, or in their capacity as IETF chairs? I.e., do they not work for other organizations while performing these roles?

I was hoping we could use this metric to identify those email domains that are associated with organizations (Google, Oracle, Microsoft) with institutional involvement in IETF from individuals who have leadership positions. It does seem to be getting there.

One thing I find confusing is that the `gmail.com` and `hotmail.com` domains have a _lower_ entropy (this metric) than `google.com`. I'm confused by this. It suggests that either `google.com` is used by a much more general population than `gmail.com`, or else that the company Google has more people working in this working group (httpbisa, in this case) than people with `gmail` addresses. Which is it?

This is the top 20 domains using this metric of involvement:

```
google.com         3.140787
gmail.com          2.984459
hotmail.com        2.553237
oracle.com         2.058373
w3.org             1.758979
yahoo.com          1.744146
microsoft.com      1.699436
akamai.com         1.585919
ericsson.com       1.512764
nokia.com          1.475076
us.ibm.com         1.430280
opera.com          1.395972
maebashi-it.org    1.273028
fb.com             1.241696
apple.com          1.192743
cisco.com          1.187929
ietf.org           1.159670
chromium.org       1.112767
cloudflare.com     1.088900
csail.mit.edu      1.054920
```


It looks like this may be because the `gmail.com` emails are actually dominated by a few individuals, making it a much more skewed distribution than `google.com`, which appears to spread responsibility for engaging the working group more evenly across its team. It's quite interesting, actually. Who is Martin Thompson?
This is the notebook where I've defined the 'domain entropy' metric and computed these results.

https://github.com/datactive/bigbang/blob/master/examples/organizations/Using%20Domain%20Entropy%20to%20Identify%20Organizations.ipynb
> It looks like this may be because the `gmail.com` emails are actually dominated by a few individuals, making it a much more skewed distribution than `google.com`, which appears to spread responsibility for engaging the working group more evenly across its team. It's quite interesting, actually. Who is Martin Thompson?

Yeah, while `gmail.com` may tend to be lots of different people with different affiliations when considered across all of IETF, say, it could be dominated by prominent individuals in one particular working group. Martin is the editor of the HTTP/2 spec and works for Mozilla: https://lowentropy.net/about/
I'm currently looking at three data sources to match up organizational affiliations to an email address:

1. IETF meeting attendance records (useful, though not exhaustive, at least from recent meetings)
2. GitHub profiles (mostly not useful so far)
3. RFC/drafts authored by the person, which list addresses and affiliations (promising, at least for the most significant contributors, but haven't coded it yet)
I was wondering whether [this](https://gist.github.com/ammarshah/f5c2624d767f91a7cbdc4e54db8dd0bf) might be helpful to filter out personal contributions, and [this](https://github.com/cncf/gitdm) to find affiliations (as suggested during the IAB-AID workshop.
At the 111 hackathon, I started using this repo that collects email providers into a CSV: https://github.com/edwin-zvs/email-providers
(And I think there are going to be many slightly similar data collection projects in this category, but that one at least has been updated in the last year and has a process for adding more domains, etc.)

But yes, it definitely shouldn't be done manually!
That gitdm tool seems to have designed a nice text based format for representing company/employer relations.

It's two kinds of files: the dev.txt files with entries of the form:

username: email, email, email
    organization from (start time) until (end time)

and then co.txt with entries:

company_name:
    username: email, email, email from (start_time) until (end time)

This seems pretty good.

In general, it seems like the data we need is a bipartite graph relating organizations and individuals, where each relation has a start time and an optional end time.
https://github.com/datactive/bigbang/pull/506/commits/54ef608e104693a942d2e8d03be915bb9e1eff2d
Is this the same as #352?
What can we pull from the IETF Datatracker? Or from RFCs which list affiliations of authors/editors in the credits? Or from some other data source?

And then drop that into a dataframe or some basic format, so that it can be consumed by other code working on analysis of organizational influence/distribution within working groups.
See #25 for discussion of this.

DataTracker attendance data is a nice source for affilation and nationality data for IETF members.

An actionable way to approach this might be:
 - Develop the affiliation data format and corresponding object, which tracks individuals, their email addresses (with times/duration), their affiliations (with times/duration), their nationalities (with times/duration) (where duration is inferred from time point data).
 - Have a script for population such an object from IETF Attendance data as an example of how to do it.
 - demonstrate how this can be used to plot affiliate interactions in a notebook as per @Christovis 's visualizatiosn from IAB-AID-1

Ultimately, such an object could wrap an ORM with an actual database behind it, and/or work with the entity resolution code to better resolve (its own) organization/affiliation references.
Is this the same as #352?
What can we pull from the IETF Datatracker? Or from RFCs which list affiliations of authors/editors in the credits? Or from some other data source?

And then drop that into a dataframe or some basic format, so that it can be consumed by other code working on analysis of organizational influence/distribution within working groups.
See #25 for discussion of this.

DataTracker attendance data is a nice source for affilation and nationality data for IETF members.

An actionable way to approach this might be:
 - Develop the affiliation data format and corresponding object, which tracks individuals, their email addresses (with times/duration), their affiliations (with times/duration), their nationalities (with times/duration) (where duration is inferred from time point data).
 - Have a script for population such an object from IETF Attendance data as an example of how to do it.
 - demonstrate how this can be used to plot affiliate interactions in a notebook as per @Christovis 's visualizatiosn from IAB-AID-1

Ultimately, such an object could wrap an ORM with an actual database behind it, and/or work with the entity resolution code to better resolve (its own) organization/affiliation references.
See #88 for signature detection issue

see also #20 
Now we have `ietfdata` as an alternative data source for affiliations. This can be used for training data for machine learning or other entity resolution techniques.
These are the top 50 most frequent email domains for the senders to the working group `httpbisa`:

gmail.com              6530
mnot.net               4117
gmx.de                 3606
phk.fre                1193
treenet.co             1006
qbik.com                843
henriknordstrom.net     790
google.com              732
chromium.org            700
gbiv.com                697
ietf.org                629
microsoft.com           614
intalio.com             420
apple.com               391
shareable.org           375
opera.com               366
greenbytes.de           329
mozilla.com             329
adambarth.com           313
ducksong.com            305
gmx.net                 282
kerwin.net              266
briansmith.org          256
laposte.net             256
xpasc.com               255
cisco.com               246
w3.org                  243
belshe.com              237
redhat.com              206
iaea.org                203
lukasa.co               203
cs.tcd                  202
it.aoy                  202
twitter.com             190
haxx.se                 183
ericsson.com            161
crf.can                 159
xyzzy.cla               150
cryptonector.com        146
zinks.de                134
elisanet.fi             127
rtfm.com                122
isode.com               120
computer.org            115
osafoundation.org       109
acm.org                 108
bisonsystems.net        104
bbc.co                   99
textuality.com           99
nygren.org               97
Obviously IETF is an uncommon space compared to many in that several of these domain names are individuals I recognize, rather than companies or email services. (It's notable that Mark, one of the chairs, has almost as many emails as all gmail users combined!) 

That some prominent individuals use their own domains in their email addresses also suggests that that limitation can be addressed by accessing their affiliation through other sources (affiliation listed in authored documents, or through a specific profile in the IETF datatracker).
I wonder if we can actually leverage the semantics of TLDs --  .com, .org, .net -- for our analysis.

It's a good point that there's a 1-to-many relationship between domains and people, but that it's a very uneven mapping. That coudl be a useful cue for whether a domain is for an organization, vs. a generic email host or a single person.

Extracting affiliation <-> email mappings from the datatracker would certainly help with this.
The only issue is that it is a benefit that's restricted to the IETF domain.
It could be used for training some other classifier though.
@npdoty @nllz I wonder if you could give me your gut check on this.

I've been working on a metric to guess whether a domain name corresponds to a significant organization vs. an individual.

When you eyeball this plot, do you see an intuitive difference between the domain names at the top of the plot vs. at the bottom of the plot?

![image](https://user-images.githubusercontent.com/68752/109816144-67e67c00-7bfe-11eb-9725-aca78f1c26b5.png)

What is that intuitive difference?
Hi Seb,

Thanks for this. The way I interpret this graph is the amount of different people that email from that specific domain. 

For example: anviwalrusden.com is the personal domain of Andrew Sullivan: former IAB chair and very active long time IETF participant. Google, Oracle, Microsoft, Nokia, etc on the other end of the spectrum have a lot of different people working in IETF, and thus many different email address contributing to discussions from that domain. 

Hope this helps.

Best,

Niels 

On 03-03-2021 14:57, Sebastian Benthall wrote:
> @npdoty <https://github.com/npdoty> @nllz <https://github.com/nllz> I wonder if you could give me your gut check on this.
> 
> I've been working on a metric to guess whether a domain name corresponds to a significant organization vs. an individual.
> 
> When you eyeball this plot, do you see an intuitive difference between the domain names at the top of the plot vs. at the bottom of the plot?
> 
> image <https://user-images.githubusercontent.com/68752/109816144-67e67c00-7bfe-11eb-9725-aca78f1c26b5.png>
> 
> What is that intuitive difference?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub <https://github.com/datactive/bigbang/issues/25#issuecomment-789731775>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCWQ5Z7W647NDMTQRV7ZWTTBY537ANCNFSM4ARFTWQA>.
> 

-- 
Twitter         :   @nielstenoever
PGP fingerprint :   8D9F C567 BEE4 A431 56C4 
                    678B 08B5 A0F2 636D 68E9

Thank you. Yes, that's what I was hoping it would get at.

For these people like Andrew Sullivan and Mark Nottingham, do they use a personal domain to signify that they are participating in their personal capacity, or in their capacity as IETF chairs? I.e., do they not work for other organizations while performing these roles?

I was hoping we could use this metric to identify those email domains that are associated with organizations (Google, Oracle, Microsoft) with institutional involvement in IETF from individuals who have leadership positions. It does seem to be getting there.

One thing I find confusing is that the `gmail.com` and `hotmail.com` domains have a _lower_ entropy (this metric) than `google.com`. I'm confused by this. It suggests that either `google.com` is used by a much more general population than `gmail.com`, or else that the company Google has more people working in this working group (httpbisa, in this case) than people with `gmail` addresses. Which is it?

This is the top 20 domains using this metric of involvement:

```
google.com         3.140787
gmail.com          2.984459
hotmail.com        2.553237
oracle.com         2.058373
w3.org             1.758979
yahoo.com          1.744146
microsoft.com      1.699436
akamai.com         1.585919
ericsson.com       1.512764
nokia.com          1.475076
us.ibm.com         1.430280
opera.com          1.395972
maebashi-it.org    1.273028
fb.com             1.241696
apple.com          1.192743
cisco.com          1.187929
ietf.org           1.159670
chromium.org       1.112767
cloudflare.com     1.088900
csail.mit.edu      1.054920
```


It looks like this may be because the `gmail.com` emails are actually dominated by a few individuals, making it a much more skewed distribution than `google.com`, which appears to spread responsibility for engaging the working group more evenly across its team. It's quite interesting, actually. Who is Martin Thompson?
This is the notebook where I've defined the 'domain entropy' metric and computed these results.

https://github.com/datactive/bigbang/blob/master/examples/organizations/Using%20Domain%20Entropy%20to%20Identify%20Organizations.ipynb
> It looks like this may be because the `gmail.com` emails are actually dominated by a few individuals, making it a much more skewed distribution than `google.com`, which appears to spread responsibility for engaging the working group more evenly across its team. It's quite interesting, actually. Who is Martin Thompson?

Yeah, while `gmail.com` may tend to be lots of different people with different affiliations when considered across all of IETF, say, it could be dominated by prominent individuals in one particular working group. Martin is the editor of the HTTP/2 spec and works for Mozilla: https://lowentropy.net/about/
I'm currently looking at three data sources to match up organizational affiliations to an email address:

1. IETF meeting attendance records (useful, though not exhaustive, at least from recent meetings)
2. GitHub profiles (mostly not useful so far)
3. RFC/drafts authored by the person, which list addresses and affiliations (promising, at least for the most significant contributors, but haven't coded it yet)
I was wondering whether [this](https://gist.github.com/ammarshah/f5c2624d767f91a7cbdc4e54db8dd0bf) might be helpful to filter out personal contributions, and [this](https://github.com/cncf/gitdm) to find affiliations (as suggested during the IAB-AID workshop.
At the 111 hackathon, I started using this repo that collects email providers into a CSV: https://github.com/edwin-zvs/email-providers
(And I think there are going to be many slightly similar data collection projects in this category, but that one at least has been updated in the last year and has a process for adding more domains, etc.)

But yes, it definitely shouldn't be done manually!
That gitdm tool seems to have designed a nice text based format for representing company/employer relations.

It's two kinds of files: the dev.txt files with entries of the form:

username: email, email, email
    organization from (start time) until (end time)

and then co.txt with entries:

company_name:
    username: email, email, email from (start_time) until (end time)

This seems pretty good.

In general, it seems like the data we need is a bipartite graph relating organizations and individuals, where each relation has a start time and an optional end time.
https://github.com/datactive/bigbang/pull/506/commits/54ef608e104693a942d2e8d03be915bb9e1eff2d
See #88 for signature detection issue

see also #20 
Now we have `ietfdata` as an alternative data source for affiliations. This can be used for training data for machine learning or other entity resolution techniques.
These are the top 50 most frequent email domains for the senders to the working group `httpbisa`:

gmail.com              6530
mnot.net               4117
gmx.de                 3606
phk.fre                1193
treenet.co             1006
qbik.com                843
henriknordstrom.net     790
google.com              732
chromium.org            700
gbiv.com                697
ietf.org                629
microsoft.com           614
intalio.com             420
apple.com               391
shareable.org           375
opera.com               366
greenbytes.de           329
mozilla.com             329
adambarth.com           313
ducksong.com            305
gmx.net                 282
kerwin.net              266
briansmith.org          256
laposte.net             256
xpasc.com               255
cisco.com               246
w3.org                  243
belshe.com              237
redhat.com              206
iaea.org                203
lukasa.co               203
cs.tcd                  202
it.aoy                  202
twitter.com             190
haxx.se                 183
ericsson.com            161
crf.can                 159
xyzzy.cla               150
cryptonector.com        146
zinks.de                134
elisanet.fi             127
rtfm.com                122
isode.com               120
computer.org            115
osafoundation.org       109
acm.org                 108
bisonsystems.net        104
bbc.co                   99
textuality.com           99
nygren.org               97
Obviously IETF is an uncommon space compared to many in that several of these domain names are individuals I recognize, rather than companies or email services. (It's notable that Mark, one of the chairs, has almost as many emails as all gmail users combined!) 

That some prominent individuals use their own domains in their email addresses also suggests that that limitation can be addressed by accessing their affiliation through other sources (affiliation listed in authored documents, or through a specific profile in the IETF datatracker).
I wonder if we can actually leverage the semantics of TLDs --  .com, .org, .net -- for our analysis.

It's a good point that there's a 1-to-many relationship between domains and people, but that it's a very uneven mapping. That coudl be a useful cue for whether a domain is for an organization, vs. a generic email host or a single person.

Extracting affiliation <-> email mappings from the datatracker would certainly help with this.
The only issue is that it is a benefit that's restricted to the IETF domain.
It could be used for training some other classifier though.
@npdoty @nllz I wonder if you could give me your gut check on this.

I've been working on a metric to guess whether a domain name corresponds to a significant organization vs. an individual.

When you eyeball this plot, do you see an intuitive difference between the domain names at the top of the plot vs. at the bottom of the plot?

![image](https://user-images.githubusercontent.com/68752/109816144-67e67c00-7bfe-11eb-9725-aca78f1c26b5.png)

What is that intuitive difference?
Hi Seb,

Thanks for this. The way I interpret this graph is the amount of different people that email from that specific domain. 

For example: anviwalrusden.com is the personal domain of Andrew Sullivan: former IAB chair and very active long time IETF participant. Google, Oracle, Microsoft, Nokia, etc on the other end of the spectrum have a lot of different people working in IETF, and thus many different email address contributing to discussions from that domain. 

Hope this helps.

Best,

Niels 

On 03-03-2021 14:57, Sebastian Benthall wrote:
> @npdoty <https://github.com/npdoty> @nllz <https://github.com/nllz> I wonder if you could give me your gut check on this.
> 
> I've been working on a metric to guess whether a domain name corresponds to a significant organization vs. an individual.
> 
> When you eyeball this plot, do you see an intuitive difference between the domain names at the top of the plot vs. at the bottom of the plot?
> 
> image <https://user-images.githubusercontent.com/68752/109816144-67e67c00-7bfe-11eb-9725-aca78f1c26b5.png>
> 
> What is that intuitive difference?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub <https://github.com/datactive/bigbang/issues/25#issuecomment-789731775>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCWQ5Z7W647NDMTQRV7ZWTTBY537ANCNFSM4ARFTWQA>.
> 

-- 
Twitter         :   @nielstenoever
PGP fingerprint :   8D9F C567 BEE4 A431 56C4 
                    678B 08B5 A0F2 636D 68E9

Thank you. Yes, that's what I was hoping it would get at.

For these people like Andrew Sullivan and Mark Nottingham, do they use a personal domain to signify that they are participating in their personal capacity, or in their capacity as IETF chairs? I.e., do they not work for other organizations while performing these roles?

I was hoping we could use this metric to identify those email domains that are associated with organizations (Google, Oracle, Microsoft) with institutional involvement in IETF from individuals who have leadership positions. It does seem to be getting there.

One thing I find confusing is that the `gmail.com` and `hotmail.com` domains have a _lower_ entropy (this metric) than `google.com`. I'm confused by this. It suggests that either `google.com` is used by a much more general population than `gmail.com`, or else that the company Google has more people working in this working group (httpbisa, in this case) than people with `gmail` addresses. Which is it?

This is the top 20 domains using this metric of involvement:

```
google.com         3.140787
gmail.com          2.984459
hotmail.com        2.553237
oracle.com         2.058373
w3.org             1.758979
yahoo.com          1.744146
microsoft.com      1.699436
akamai.com         1.585919
ericsson.com       1.512764
nokia.com          1.475076
us.ibm.com         1.430280
opera.com          1.395972
maebashi-it.org    1.273028
fb.com             1.241696
apple.com          1.192743
cisco.com          1.187929
ietf.org           1.159670
chromium.org       1.112767
cloudflare.com     1.088900
csail.mit.edu      1.054920
```


It looks like this may be because the `gmail.com` emails are actually dominated by a few individuals, making it a much more skewed distribution than `google.com`, which appears to spread responsibility for engaging the working group more evenly across its team. It's quite interesting, actually. Who is Martin Thompson?
This is the notebook where I've defined the 'domain entropy' metric and computed these results.

https://github.com/datactive/bigbang/blob/master/examples/organizations/Using%20Domain%20Entropy%20to%20Identify%20Organizations.ipynb
> It looks like this may be because the `gmail.com` emails are actually dominated by a few individuals, making it a much more skewed distribution than `google.com`, which appears to spread responsibility for engaging the working group more evenly across its team. It's quite interesting, actually. Who is Martin Thompson?

Yeah, while `gmail.com` may tend to be lots of different people with different affiliations when considered across all of IETF, say, it could be dominated by prominent individuals in one particular working group. Martin is the editor of the HTTP/2 spec and works for Mozilla: https://lowentropy.net/about/
I'm currently looking at three data sources to match up organizational affiliations to an email address:

1. IETF meeting attendance records (useful, though not exhaustive, at least from recent meetings)
2. GitHub profiles (mostly not useful so far)
3. RFC/drafts authored by the person, which list addresses and affiliations (promising, at least for the most significant contributors, but haven't coded it yet)
I was wondering whether [this](https://gist.github.com/ammarshah/f5c2624d767f91a7cbdc4e54db8dd0bf) might be helpful to filter out personal contributions, and [this](https://github.com/cncf/gitdm) to find affiliations (as suggested during the IAB-AID workshop.
At the 111 hackathon, I started using this repo that collects email providers into a CSV: https://github.com/edwin-zvs/email-providers
(And I think there are going to be many slightly similar data collection projects in this category, but that one at least has been updated in the last year and has a process for adding more domains, etc.)

But yes, it definitely shouldn't be done manually!
That gitdm tool seems to have designed a nice text based format for representing company/employer relations.

It's two kinds of files: the dev.txt files with entries of the form:

username: email, email, email
    organization from (start time) until (end time)

and then co.txt with entries:

company_name:
    username: email, email, email from (start_time) until (end time)

This seems pretty good.

In general, it seems like the data we need is a bipartite graph relating organizations and individuals, where each relation has a start time and an optional end time.
https://github.com/datactive/bigbang/pull/506/commits/54ef608e104693a942d2e8d03be915bb9e1eff2d
See #88 for signature detection issue

see also #20 
Now we have `ietfdata` as an alternative data source for affiliations. This can be used for training data for machine learning or other entity resolution techniques.
These are the top 50 most frequent email domains for the senders to the working group `httpbisa`:

gmail.com              6530
mnot.net               4117
gmx.de                 3606
phk.fre                1193
treenet.co             1006
qbik.com                843
henriknordstrom.net     790
google.com              732
chromium.org            700
gbiv.com                697
ietf.org                629
microsoft.com           614
intalio.com             420
apple.com               391
shareable.org           375
opera.com               366
greenbytes.de           329
mozilla.com             329
adambarth.com           313
ducksong.com            305
gmx.net                 282
kerwin.net              266
briansmith.org          256
laposte.net             256
xpasc.com               255
cisco.com               246
w3.org                  243
belshe.com              237
redhat.com              206
iaea.org                203
lukasa.co               203
cs.tcd                  202
it.aoy                  202
twitter.com             190
haxx.se                 183
ericsson.com            161
crf.can                 159
xyzzy.cla               150
cryptonector.com        146
zinks.de                134
elisanet.fi             127
rtfm.com                122
isode.com               120
computer.org            115
osafoundation.org       109
acm.org                 108
bisonsystems.net        104
bbc.co                   99
textuality.com           99
nygren.org               97
Obviously IETF is an uncommon space compared to many in that several of these domain names are individuals I recognize, rather than companies or email services. (It's notable that Mark, one of the chairs, has almost as many emails as all gmail users combined!) 

That some prominent individuals use their own domains in their email addresses also suggests that that limitation can be addressed by accessing their affiliation through other sources (affiliation listed in authored documents, or through a specific profile in the IETF datatracker).
I wonder if we can actually leverage the semantics of TLDs --  .com, .org, .net -- for our analysis.

It's a good point that there's a 1-to-many relationship between domains and people, but that it's a very uneven mapping. That coudl be a useful cue for whether a domain is for an organization, vs. a generic email host or a single person.

Extracting affiliation <-> email mappings from the datatracker would certainly help with this.
The only issue is that it is a benefit that's restricted to the IETF domain.
It could be used for training some other classifier though.
@npdoty @nllz I wonder if you could give me your gut check on this.

I've been working on a metric to guess whether a domain name corresponds to a significant organization vs. an individual.

When you eyeball this plot, do you see an intuitive difference between the domain names at the top of the plot vs. at the bottom of the plot?

![image](https://user-images.githubusercontent.com/68752/109816144-67e67c00-7bfe-11eb-9725-aca78f1c26b5.png)

What is that intuitive difference?
Hi Seb,

Thanks for this. The way I interpret this graph is the amount of different people that email from that specific domain. 

For example: anviwalrusden.com is the personal domain of Andrew Sullivan: former IAB chair and very active long time IETF participant. Google, Oracle, Microsoft, Nokia, etc on the other end of the spectrum have a lot of different people working in IETF, and thus many different email address contributing to discussions from that domain. 

Hope this helps.

Best,

Niels 

On 03-03-2021 14:57, Sebastian Benthall wrote:
> @npdoty <https://github.com/npdoty> @nllz <https://github.com/nllz> I wonder if you could give me your gut check on this.
> 
> I've been working on a metric to guess whether a domain name corresponds to a significant organization vs. an individual.
> 
> When you eyeball this plot, do you see an intuitive difference between the domain names at the top of the plot vs. at the bottom of the plot?
> 
> image <https://user-images.githubusercontent.com/68752/109816144-67e67c00-7bfe-11eb-9725-aca78f1c26b5.png>
> 
> What is that intuitive difference?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub <https://github.com/datactive/bigbang/issues/25#issuecomment-789731775>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCWQ5Z7W647NDMTQRV7ZWTTBY537ANCNFSM4ARFTWQA>.
> 

-- 
Twitter         :   @nielstenoever
PGP fingerprint :   8D9F C567 BEE4 A431 56C4 
                    678B 08B5 A0F2 636D 68E9

Thank you. Yes, that's what I was hoping it would get at.

For these people like Andrew Sullivan and Mark Nottingham, do they use a personal domain to signify that they are participating in their personal capacity, or in their capacity as IETF chairs? I.e., do they not work for other organizations while performing these roles?

I was hoping we could use this metric to identify those email domains that are associated with organizations (Google, Oracle, Microsoft) with institutional involvement in IETF from individuals who have leadership positions. It does seem to be getting there.

One thing I find confusing is that the `gmail.com` and `hotmail.com` domains have a _lower_ entropy (this metric) than `google.com`. I'm confused by this. It suggests that either `google.com` is used by a much more general population than `gmail.com`, or else that the company Google has more people working in this working group (httpbisa, in this case) than people with `gmail` addresses. Which is it?

This is the top 20 domains using this metric of involvement:

```
google.com         3.140787
gmail.com          2.984459
hotmail.com        2.553237
oracle.com         2.058373
w3.org             1.758979
yahoo.com          1.744146
microsoft.com      1.699436
akamai.com         1.585919
ericsson.com       1.512764
nokia.com          1.475076
us.ibm.com         1.430280
opera.com          1.395972
maebashi-it.org    1.273028
fb.com             1.241696
apple.com          1.192743
cisco.com          1.187929
ietf.org           1.159670
chromium.org       1.112767
cloudflare.com     1.088900
csail.mit.edu      1.054920
```


It looks like this may be because the `gmail.com` emails are actually dominated by a few individuals, making it a much more skewed distribution than `google.com`, which appears to spread responsibility for engaging the working group more evenly across its team. It's quite interesting, actually. Who is Martin Thompson?
This is the notebook where I've defined the 'domain entropy' metric and computed these results.

https://github.com/datactive/bigbang/blob/master/examples/organizations/Using%20Domain%20Entropy%20to%20Identify%20Organizations.ipynb
> It looks like this may be because the `gmail.com` emails are actually dominated by a few individuals, making it a much more skewed distribution than `google.com`, which appears to spread responsibility for engaging the working group more evenly across its team. It's quite interesting, actually. Who is Martin Thompson?

Yeah, while `gmail.com` may tend to be lots of different people with different affiliations when considered across all of IETF, say, it could be dominated by prominent individuals in one particular working group. Martin is the editor of the HTTP/2 spec and works for Mozilla: https://lowentropy.net/about/
I'm currently looking at three data sources to match up organizational affiliations to an email address:

1. IETF meeting attendance records (useful, though not exhaustive, at least from recent meetings)
2. GitHub profiles (mostly not useful so far)
3. RFC/drafts authored by the person, which list addresses and affiliations (promising, at least for the most significant contributors, but haven't coded it yet)
I was wondering whether [this](https://gist.github.com/ammarshah/f5c2624d767f91a7cbdc4e54db8dd0bf) might be helpful to filter out personal contributions, and [this](https://github.com/cncf/gitdm) to find affiliations (as suggested during the IAB-AID workshop.
At the 111 hackathon, I started using this repo that collects email providers into a CSV: https://github.com/edwin-zvs/email-providers
(And I think there are going to be many slightly similar data collection projects in this category, but that one at least has been updated in the last year and has a process for adding more domains, etc.)

But yes, it definitely shouldn't be done manually!
That gitdm tool seems to have designed a nice text based format for representing company/employer relations.

It's two kinds of files: the dev.txt files with entries of the form:

username: email, email, email
    organization from (start time) until (end time)

and then co.txt with entries:

company_name:
    username: email, email, email from (start_time) until (end time)

This seems pretty good.

In general, it seems like the data we need is a bipartite graph relating organizations and individuals, where each relation has a start time and an optional end time.
https://github.com/datactive/bigbang/pull/506/commits/54ef608e104693a942d2e8d03be915bb9e1eff2d
See #88 for signature detection issue

see also #20 
Now we have `ietfdata` as an alternative data source for affiliations. This can be used for training data for machine learning or other entity resolution techniques.
These are the top 50 most frequent email domains for the senders to the working group `httpbisa`:

gmail.com              6530
mnot.net               4117
gmx.de                 3606
phk.fre                1193
treenet.co             1006
qbik.com                843
henriknordstrom.net     790
google.com              732
chromium.org            700
gbiv.com                697
ietf.org                629
microsoft.com           614
intalio.com             420
apple.com               391
shareable.org           375
opera.com               366
greenbytes.de           329
mozilla.com             329
adambarth.com           313
ducksong.com            305
gmx.net                 282
kerwin.net              266
briansmith.org          256
laposte.net             256
xpasc.com               255
cisco.com               246
w3.org                  243
belshe.com              237
redhat.com              206
iaea.org                203
lukasa.co               203
cs.tcd                  202
it.aoy                  202
twitter.com             190
haxx.se                 183
ericsson.com            161
crf.can                 159
xyzzy.cla               150
cryptonector.com        146
zinks.de                134
elisanet.fi             127
rtfm.com                122
isode.com               120
computer.org            115
osafoundation.org       109
acm.org                 108
bisonsystems.net        104
bbc.co                   99
textuality.com           99
nygren.org               97
Obviously IETF is an uncommon space compared to many in that several of these domain names are individuals I recognize, rather than companies or email services. (It's notable that Mark, one of the chairs, has almost as many emails as all gmail users combined!) 

That some prominent individuals use their own domains in their email addresses also suggests that that limitation can be addressed by accessing their affiliation through other sources (affiliation listed in authored documents, or through a specific profile in the IETF datatracker).
I wonder if we can actually leverage the semantics of TLDs --  .com, .org, .net -- for our analysis.

It's a good point that there's a 1-to-many relationship between domains and people, but that it's a very uneven mapping. That coudl be a useful cue for whether a domain is for an organization, vs. a generic email host or a single person.

Extracting affiliation <-> email mappings from the datatracker would certainly help with this.
The only issue is that it is a benefit that's restricted to the IETF domain.
It could be used for training some other classifier though.
@npdoty @nllz I wonder if you could give me your gut check on this.

I've been working on a metric to guess whether a domain name corresponds to a significant organization vs. an individual.

When you eyeball this plot, do you see an intuitive difference between the domain names at the top of the plot vs. at the bottom of the plot?

![image](https://user-images.githubusercontent.com/68752/109816144-67e67c00-7bfe-11eb-9725-aca78f1c26b5.png)

What is that intuitive difference?
Hi Seb,

Thanks for this. The way I interpret this graph is the amount of different people that email from that specific domain. 

For example: anviwalrusden.com is the personal domain of Andrew Sullivan: former IAB chair and very active long time IETF participant. Google, Oracle, Microsoft, Nokia, etc on the other end of the spectrum have a lot of different people working in IETF, and thus many different email address contributing to discussions from that domain. 

Hope this helps.

Best,

Niels 

On 03-03-2021 14:57, Sebastian Benthall wrote:
> @npdoty <https://github.com/npdoty> @nllz <https://github.com/nllz> I wonder if you could give me your gut check on this.
> 
> I've been working on a metric to guess whether a domain name corresponds to a significant organization vs. an individual.
> 
> When you eyeball this plot, do you see an intuitive difference between the domain names at the top of the plot vs. at the bottom of the plot?
> 
> image <https://user-images.githubusercontent.com/68752/109816144-67e67c00-7bfe-11eb-9725-aca78f1c26b5.png>
> 
> What is that intuitive difference?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub <https://github.com/datactive/bigbang/issues/25#issuecomment-789731775>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCWQ5Z7W647NDMTQRV7ZWTTBY537ANCNFSM4ARFTWQA>.
> 

-- 
Twitter         :   @nielstenoever
PGP fingerprint :   8D9F C567 BEE4 A431 56C4 
                    678B 08B5 A0F2 636D 68E9

Thank you. Yes, that's what I was hoping it would get at.

For these people like Andrew Sullivan and Mark Nottingham, do they use a personal domain to signify that they are participating in their personal capacity, or in their capacity as IETF chairs? I.e., do they not work for other organizations while performing these roles?

I was hoping we could use this metric to identify those email domains that are associated with organizations (Google, Oracle, Microsoft) with institutional involvement in IETF from individuals who have leadership positions. It does seem to be getting there.

One thing I find confusing is that the `gmail.com` and `hotmail.com` domains have a _lower_ entropy (this metric) than `google.com`. I'm confused by this. It suggests that either `google.com` is used by a much more general population than `gmail.com`, or else that the company Google has more people working in this working group (httpbisa, in this case) than people with `gmail` addresses. Which is it?

This is the top 20 domains using this metric of involvement:

```
google.com         3.140787
gmail.com          2.984459
hotmail.com        2.553237
oracle.com         2.058373
w3.org             1.758979
yahoo.com          1.744146
microsoft.com      1.699436
akamai.com         1.585919
ericsson.com       1.512764
nokia.com          1.475076
us.ibm.com         1.430280
opera.com          1.395972
maebashi-it.org    1.273028
fb.com             1.241696
apple.com          1.192743
cisco.com          1.187929
ietf.org           1.159670
chromium.org       1.112767
cloudflare.com     1.088900
csail.mit.edu      1.054920
```


It looks like this may be because the `gmail.com` emails are actually dominated by a few individuals, making it a much more skewed distribution than `google.com`, which appears to spread responsibility for engaging the working group more evenly across its team. It's quite interesting, actually. Who is Martin Thompson?
This is the notebook where I've defined the 'domain entropy' metric and computed these results.

https://github.com/datactive/bigbang/blob/master/examples/organizations/Using%20Domain%20Entropy%20to%20Identify%20Organizations.ipynb
> It looks like this may be because the `gmail.com` emails are actually dominated by a few individuals, making it a much more skewed distribution than `google.com`, which appears to spread responsibility for engaging the working group more evenly across its team. It's quite interesting, actually. Who is Martin Thompson?

Yeah, while `gmail.com` may tend to be lots of different people with different affiliations when considered across all of IETF, say, it could be dominated by prominent individuals in one particular working group. Martin is the editor of the HTTP/2 spec and works for Mozilla: https://lowentropy.net/about/
I'm currently looking at three data sources to match up organizational affiliations to an email address:

1. IETF meeting attendance records (useful, though not exhaustive, at least from recent meetings)
2. GitHub profiles (mostly not useful so far)
3. RFC/drafts authored by the person, which list addresses and affiliations (promising, at least for the most significant contributors, but haven't coded it yet)
I was wondering whether [this](https://gist.github.com/ammarshah/f5c2624d767f91a7cbdc4e54db8dd0bf) might be helpful to filter out personal contributions, and [this](https://github.com/cncf/gitdm) to find affiliations (as suggested during the IAB-AID workshop.
At the 111 hackathon, I started using this repo that collects email providers into a CSV: https://github.com/edwin-zvs/email-providers
(And I think there are going to be many slightly similar data collection projects in this category, but that one at least has been updated in the last year and has a process for adding more domains, etc.)

But yes, it definitely shouldn't be done manually!
That gitdm tool seems to have designed a nice text based format for representing company/employer relations.

It's two kinds of files: the dev.txt files with entries of the form:

username: email, email, email
    organization from (start time) until (end time)

and then co.txt with entries:

company_name:
    username: email, email, email from (start_time) until (end time)

This seems pretty good.

In general, it seems like the data we need is a bipartite graph relating organizations and individuals, where each relation has a start time and an optional end time.
https://github.com/datactive/bigbang/pull/506/commits/54ef608e104693a942d2e8d03be915bb9e1eff2d

Fix patch:
diff --git a/bigbang/analysis/attendance.py b/bigbang/analysis/attendance.py
new file mode 100644
index 00000000..a33b400c
--- /dev/null
+++ b/bigbang/analysis/attendance.py
@@ -0,0 +1,167 @@
+import bigbang.analysis.process as process
+import bigbang.analysis.utils as utils
+import dataclasses
+from ietfdata.datatracker import DataTracker, MeetingTypeURI
+from ietfdata.datatracker_ext import DataTrackerExt
+import pandas as pd
+
+
+def name_email_affil_relations_from_IETF_attendance(
+    meeting_range=[106, 107, 108], threshold=None
+):
+    """
+
+    Extract and infer from IETF attendance records relations between full names, email address, and affiliations.
+
+    In the returned dataframes, each row represents a relation between two of these forms of entity,
+    along with the maximum and minimum date associated with it in the data.
+
+    Two forms of inference are used when generating these relational tables:
+
+     - Missing values in time are filled forward, then filled backward
+     - TODO: Affiliations are ran through the entity resolution script to reduce them to a 'cannonical form'
+
+    Parameters
+    ------------
+
+    meeting_range: list of ints
+        The numbers of the IETF meetings to use for source data
+
+    threshold: float
+        Defaults to None. If not None, activate entity resolution on the affiliations.
+        Threshold value is used for the entity resolution.
+
+
+    Returns
+    -----------
+
+    rel_name_affil: pandas.DataFrame
+
+    rel_email_affil: pandas.DataFrame
+
+    rel_name_email: pandas.DataFrame
+
+
+    """
+    datatracker = DataTracker()
+    dt = DataTrackerExt()  # initialize, for all meeting registration downloads
+
+    meetings = datatracker.meetings(
+        meeting_type=datatracker.meeting_type(
+            MeetingTypeURI("/api/v1/name/meetingtypename/ietf/")
+        )
+    )
+    full_ietf_meetings = list(meetings)
+
+    meeting_attendees_df = pd.DataFrame()
+    ietf_meetings = []
+    for meeting in full_ietf_meetings:
+        meetingd = dataclasses.asdict(meeting)
+        meetingd["meeting_obj"] = meeting
+        meetingd["num"] = int(meeting.number)
+        ietf_meetings.append(meetingd)
+
+    for meeting in ietf_meetings:
+        if (
+            meeting["num"] in meeting_range
+        ):  # can filter here by the meetings to analyze
+            registrations = dt.meeting_registrations(
+                meeting=meeting["meeting_obj"]
+            )
+            df = pd.DataFrame.from_records(
+                [dataclasses.asdict(x) for x in list(registrations)]
+            )
+            df["num"] = meeting["num"]
+            df["date"] = meeting["date"]
+            df["domain"] = df["email"].apply(utils.extract_domain)
+            full_name = df["first_name"] + " " + df["last_name"]
+            df["full_name"] = full_name
+            meeting_attendees_df = meeting_attendees_df.append(df)
+
+    ind_affiliation = meeting_attendees_df[
+        ["full_name", "affiliation", "email", "domain", "date"]
+    ]
+
+    if threshold is not None:
+        org_names = ind_affiliation["affiliation"].dropna().value_counts()
+        ents = process.resolve_entities(
+            org_names, process.containment_distance, threshold=0.15
+        )
+
+        replacements = {}
+        for r in [{name: ent for name in ents[ent]} for ent in ents]:
+            replacements.update(r)
+
+        ind_affiliation.loc[:, "affiliation"] = ind_affiliation[
+            "affiliation"
+        ].apply(
+            lambda x: replacements[x]
+        )  # , axis = 1)
+
+    # full_name / affiliation
+    name_affil_dates = (
+        ind_affiliation.pivot_table(
+            index="date",
+            columns="full_name",
+            values="affiliation",
+            aggfunc="first",
+        )
+        .fillna(method="ffill")
+        .fillna(method="bfill")
+    )
+
+    long_name_affils = name_affil_dates.reset_index().melt(
+        id_vars=["date"], value_name="affiliation"
+    )
+
+    rel_name_affil = (
+        long_name_affils.groupby(["full_name", "affiliation"])["date"]
+        .agg(["min", "max"])
+        .reset_index()
+        .rename({"min": "min_date", "max": "max_date"}, axis=1)
+    )
+
+    # email / affiliation
+    email_affil_dates = (
+        ind_affiliation.pivot_table(
+            index="date",
+            columns="email",
+            values="affiliation",
+            aggfunc="first",
+        )
+        .fillna(method="ffill")
+        .fillna(method="bfill")
+    )
+
+    long_email_affils = email_affil_dates.reset_index().melt(
+        id_vars=["date"], value_name="affiliation"
+    )
+
+    rel_email_affil = (
+        long_email_affils.groupby(["email", "affiliation"])["date"]
+        .agg(["min", "max"])
+        .reset_index()
+        .rename({"min": "min_date", "max": "max_date"}, axis=1)
+    )
+
+    # name / email
+    name_email_dates = (
+        ind_affiliation.pivot_table(
+            index="date", columns="full_name", values="email", aggfunc="first"
+        )
+        .fillna(method="ffill")
+        .fillna(method="bfill")
+    )
+
+    long_name_email = name_email_dates.reset_index().melt(
+        id_vars=["date"], value_name="email"
+    )
+
+    rel_name_email = (
+        long_name_email.groupby(["full_name", "email"])["date"]
+        .agg(["min", "max"])
+        .reset_index()
+        .rename({"min": "min_date", "max": "max_date"}, axis=1)
+    )
+
+    return rel_name_affil, rel_email_affil, rel_name_email
diff --git a/bigbang/analysis/repo_loader.py b/bigbang/analysis/repo_loader.py
index a0a1c624..be7ad75c 100644
--- a/bigbang/analysis/repo_loader.py
+++ b/bigbang/analysis/repo_loader.py
@@ -9,8 +9,8 @@
 import networkx as nx
 import pandas as pd
 import requests
-from IPython.nbconvert import PythonExporter
-from IPython.nbformat import current as nbformat
+from nbconvert import PythonExporter
+from nbformat import current as nbformat
 
 from config.config import CONFIG
 
diff --git a/bigbang/archive.py b/bigbang/archive.py
index eac2fd07..6093db55 100644
--- a/bigbang/archive.py
+++ b/bigbang/archive.py
@@ -163,6 +163,49 @@ def __init__(self, data, archive_dir=CONFIG.mail_path, mbox=False):
                 "Archive after initial processing is empty. Was data collected properly?"
             )
 
+    def add_affiliation(self, rel_email_affil):
+        """
+        Uses a DataFrame of email affiliation information and adds it to the archive's data table.
+
+        The email affilation data is expected to have a regular format, with columns:
+
+          - ``email`` - strings, complete email addresses
+          - ``affilation`` - strings, names of organizatiosn of affilation
+          - ``min_date`` - datetime, the starting date of the affiliation
+          - ``max_date`` - datetime,the end date of the affilation.
+
+        Note that this mutates the dataframe in ``self.data`` to add the affiliation data.
+
+        Params
+        ------
+
+        rel_email_affil : pandas.DataFrame
+        """
+
+        def match_affiliation(row):
+            ## TODO: Option to turn off the max or min constraints in case of limited data.
+            matches = rel_email_affil[
+                (
+                    rel_email_affil["email"]
+                    == analysis_utils.extract_email(row["From"])
+                )
+                & (
+                    rel_email_affil["min_date"].dt.tz_localize("utc")
+                    <= pd.to_datetime(row["Date"])
+                )
+                & (
+                    rel_email_affil["max_date"].dt.tz_localize("utc")
+                    >= pd.to_datetime(row["Date"])
+                )
+            ]
+            return (
+                matches["affiliation"].values[0]
+                if matches.shape[0] > 0
+                else None
+            )
+
+        self.data["affiliation"] = self.data.apply(match_affiliation, axis=1)
+
     def resolve_entities(self, inplace=True):
         """Return data with resolved entities.
 
diff --git a/examples/attendance/Extracting Org-Domain and Person-Org-Duration Information From Attendence Data.ipynb b/examples/attendance/Extracting Org-Domain and Person-Org-Duration Information From Attendance Data.ipynb
similarity index 56%
rename from examples/attendance/Extracting Org-Domain and Person-Org-Duration Information From Attendence Data.ipynb
rename to examples/attendance/Extracting Org-Domain and Person-Org-Duration Information From Attendance Data.ipynb
index 3b6ad3d2..ea98eae0 100644
--- a/examples/attendance/Extracting Org-Domain and Person-Org-Duration Information From Attendence Data.ipynb
+++ b/examples/attendance/Extracting Org-Domain and Person-Org-Duration Information From Attendance Data.ipynb
@@ -13,8 +13,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import bigbang.domain as domain\n",
     "import bigbang.datasets.domains as domains\n",
+    "import bigbang.analysis.utils as utils\n",
+    "import bigbang.analysis.attendance as attendance\n",
     "\n",
     "from ietfdata.datatracker     import *\n",
     "from ietfdata.datatracker_ext import *\n",
@@ -29,14 +30,14 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "org_cats = pd.read_csv(\"organization_categories - organization_categories.csv\")"
+    "org_cats = pd.read_csv(\"../organizations/organization_categories.csv\")"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Getting attendence records from datatracker\n",
+    "## Getting attendance records from datatracker\n",
     "\n",
     "When attendees register for a meeting, the report their name, email address, and affiliation.\n",
     "\n",
@@ -49,15 +50,7 @@
    "cell_type": "code",
    "execution_count": 3,
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "INFO:ietfdata:glasgow-ietfdata/0.5.1 (cache disabled)\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "datatracker = DataTracker()\n",
     "\n",
@@ -99,15 +92,7 @@
    "cell_type": "code",
    "execution_count": 6,
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "INFO:ietfdata:glasgow-ietfdata/0.5.1 (cache disabled)\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "dt = DataTrackerExt() # initialize, for all meeting registration downloads"
    ]
@@ -152,7 +137,7 @@
     "        df = pd.DataFrame.from_records([dataclasses.asdict(x) for x in list(registrations)])\n",
     "        df['num'] = meeting['num']\n",
     "        df['date'] = meeting['date']\n",
-    "        df['domain'] = df['email'].apply(domain.extract_domain)\n",
+    "        df['domain'] = df['email'].apply(utils.extract_domain)\n",
     "        full_name = df['first_name'] + \" \" + df['last_name']\n",
     "        df['full_name'] = full_name\n",
     "        meeting_attendees_df = meeting_attendees_df.append(df)"
@@ -334,6 +319,37 @@
     "ind_affiliation[:10]"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "                                                          1071\n",
+       "Cisco                                                      166\n",
+       "Google                                                     151\n",
+       "Huawei                                                     149\n",
+       "Cisco Systems                                              140\n",
+       "                                                          ... \n",
+       "ISAE-SUPAERO / TESA                                          1\n",
+       "Good API                                                     1\n",
+       "Towson University                                            1\n",
+       "NUS                                                          1\n",
+       "<a href='https://ghanarecruitment.com'>Recruitment</a>       1\n",
+       "Name: affiliation, Length: 1860, dtype: int64"
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "ind_affiliation['affiliation'].dropna().value_counts()"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -343,7 +359,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 12,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -360,7 +376,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": 13,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -372,7 +388,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 13,
+   "execution_count": 14,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -382,7 +398,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 14,
+   "execution_count": 15,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -392,7 +408,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 15,
+   "execution_count": 16,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -413,7 +429,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 16,
+   "execution_count": 17,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -422,7 +438,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 17,
+   "execution_count": 18,
    "metadata": {},
    "outputs": [
     {
@@ -573,7 +589,7 @@
        "[100 rows x 5 columns]"
       ]
      },
-     "execution_count": 17,
+     "execution_count": 18,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -584,13 +600,45 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 18,
+   "execution_count": 19,
    "metadata": {},
    "outputs": [],
    "source": [
     "ad_stats[:100].to_csv(\"affiliation_domain_stats.csv\")"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "affiliation\n",
+       "Huawei                  135.0\n",
+       "Cisco                   139.0\n",
+       "Cisco Systems           128.0\n",
+       "Google                  130.0\n",
+       "Ericsson                125.0\n",
+       "                        ...  \n",
+       "windows                   0.0\n",
+       "www Safegnamain. com      0.0\n",
+       "xierqi                    0.0\n",
+       "ابراهيم                   0.0\n",
+       "恩典之前  讚美之裏                0.0\n",
+       "Name: sum, Length: 1860, dtype: float64"
+      ]
+     },
+     "execution_count": 20,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "ad_stats['sum']"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -606,7 +654,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 19,
+   "execution_count": 21,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -620,7 +668,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 20,
+   "execution_count": 22,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -629,7 +677,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 21,
+   "execution_count": 23,
    "metadata": {},
    "outputs": [
     {
@@ -649,7 +697,7 @@
        "      dtype='object', name='full_name')"
       ]
      },
-     "execution_count": 21,
+     "execution_count": 23,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -660,7 +708,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 22,
+   "execution_count": 24,
    "metadata": {},
    "outputs": [
     {
@@ -948,7 +996,7 @@
        "[6 rows x 40 columns]"
       ]
      },
-     "execution_count": 22,
+     "execution_count": 24,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -959,7 +1007,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 23,
+   "execution_count": 25,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -975,16 +1023,16 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 24,
+   "execution_count": 26,
    "metadata": {},
    "outputs": [],
    "source": [
-    "import bigbang.process as process"
+    "import bigbang.analysis.process as process"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 25,
+   "execution_count": 27,
    "metadata": {},
    "outputs": [
     {
@@ -1024,7 +1072,7 @@
        "      <td>3GPP</td>\n",
        "      <td>Standards Body</td>\n",
        "      <td>NaN</td>\n",
-       "      <td>Governance body</td>\n",
+       "      <td>Technical Community</td>\n",
        "      <td>NaN</td>\n",
        "      <td>3gpp.org</td>\n",
        "      <td>NaN</td>\n",
@@ -1068,7 +1116,7 @@
        "      <td>5G Automotive Association</td>\n",
        "      <td>Association</td>\n",
        "      <td>NaN</td>\n",
-       "      <td>NaN</td>\n",
+       "      <td>Business</td>\n",
        "      <td>Germany</td>\n",
        "      <td>NaN</td>\n",
        "      <td>OTHER</td>\n",
@@ -1086,7 +1134,7 @@
        "      <td>...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>902</th>\n",
+       "      <th>938</th>\n",
        "      <td>Zhejiang University</td>\n",
        "      <td>Research Institution</td>\n",
        "      <td>NaN</td>\n",
@@ -1097,40 +1145,40 @@
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>903</th>\n",
+       "      <th>939</th>\n",
        "      <td>Zollkriminalamt (ZKA)</td>\n",
        "      <td>NaN</td>\n",
        "      <td>NaN</td>\n",
-       "      <td>NaN</td>\n",
+       "      <td>Government</td>\n",
        "      <td>Germany</td>\n",
        "      <td>NaN</td>\n",
        "      <td>ETSI</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>904</th>\n",
+       "      <th>940</th>\n",
        "      <td>ZONSON SAMRT AUTO CORPORATION</td>\n",
+       "      <td>Consumer hardware and software vendor</td>\n",
        "      <td>NaN</td>\n",
-       "      <td>NaN</td>\n",
-       "      <td>NaN</td>\n",
+       "      <td>Business</td>\n",
        "      <td>China</td>\n",
        "      <td>NaN</td>\n",
        "      <td>CCSA</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>905</th>\n",
+       "      <th>941</th>\n",
        "      <td>ZTE</td>\n",
        "      <td>Networking equipment vendor</td>\n",
        "      <td>NaN</td>\n",
        "      <td>Business</td>\n",
        "      <td>China</td>\n",
-       "      <td>zte.com.cn , zte.com</td>\n",
+       "      <td>zte.com.cn, zte.com</td>\n",
        "      <td>CCSA</td>\n",
        "      <td>NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>909</th>\n",
+       "      <th>945</th>\n",
        "      <td>ZXNE CORPORATION</td>\n",
        "      <td>NaN</td>\n",
        "      <td>NaN</td>\n",
@@ -1142,53 +1190,53 @@
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
-       "<p>666 rows × 8 columns</p>\n",
+       "<p>682 rows × 8 columns</p>\n",
        "</div>"
       ],
       "text/plain": [
-       "                              name                     category  \\\n",
-       "0                             3GPP               Standards Body   \n",
-       "1                  450connect GmbH                          NaN   \n",
-       "2                      4G Americas                          NaN   \n",
-       "3                      5G Americas                          NaN   \n",
-       "4        5G Automotive Association                  Association   \n",
-       "..                             ...                          ...   \n",
-       "902            Zhejiang University         Research Institution   \n",
-       "903          Zollkriminalamt (ZKA)                          NaN   \n",
-       "904  ZONSON SAMRT AUTO CORPORATION                          NaN   \n",
-       "905                            ZTE  Networking equipment vendor   \n",
-       "909               ZXNE CORPORATION                          NaN   \n",
+       "                              name                               category  \\\n",
+       "0                             3GPP                         Standards Body   \n",
+       "1                  450connect GmbH                                    NaN   \n",
+       "2                      4G Americas                                    NaN   \n",
+       "3                      5G Americas                                    NaN   \n",
+       "4        5G Automotive Association                            Association   \n",
+       "..                             ...                                    ...   \n",
+       "938            Zhejiang University                   Research Institution   \n",
+       "939          Zollkriminalamt (ZKA)                                    NaN   \n",
+       "940  ZONSON SAMRT AUTO CORPORATION  Consumer hardware and software vendor   \n",
+       "941                            ZTE            Networking equipment vendor   \n",
+       "945               ZXNE CORPORATION                                    NaN   \n",
        "\n",
-       "    subsidiary of / alias of stakeholdergroup    nationality  \\\n",
-       "0                        NaN  Governance body            NaN   \n",
-       "1                        NaN              NaN        Germany   \n",
-       "2                        NaN              NaN  United States   \n",
-       "3                        NaN              NaN  United States   \n",
-       "4                        NaN              NaN        Germany   \n",
-       "..                       ...              ...            ...   \n",
-       "902                      NaN         Academia          China   \n",
-       "903                      NaN              NaN        Germany   \n",
-       "904                      NaN              NaN          China   \n",
-       "905                      NaN         Business          China   \n",
-       "909                      NaN              NaN          China   \n",
+       "    subsidiary of / alias of     stakeholdergroup    nationality  \\\n",
+       "0                        NaN  Technical Community            NaN   \n",
+       "1                        NaN                  NaN        Germany   \n",
+       "2                        NaN                  NaN  United States   \n",
+       "3                        NaN                  NaN  United States   \n",
+       "4                        NaN             Business        Germany   \n",
+       "..                       ...                  ...            ...   \n",
+       "938                      NaN             Academia          China   \n",
+       "939                      NaN           Government        Germany   \n",
+       "940                      NaN             Business          China   \n",
+       "941                      NaN             Business          China   \n",
+       "945                      NaN                  NaN          China   \n",
        "\n",
-       "       email domain names Membership Organization Wiki Page  \n",
-       "0                3gpp.org                     NaN       NaN  \n",
-       "1                     NaN                    ETSI       NaN  \n",
-       "2                     NaN                   OTHER       NaN  \n",
-       "3                     NaN                   OTHER       NaN  \n",
-       "4                     NaN                   OTHER       NaN  \n",
-       "..                    ...                     ...       ...  \n",
-       "902                   NaN                    CCSA       NaN  \n",
-       "903                   NaN                    ETSI       NaN  \n",
-       "904                   NaN                    CCSA       NaN  \n",
-       "905  zte.com.cn , zte.com                    CCSA       NaN  \n",
-       "909                   NaN                    CCSA       NaN  \n",
+       "      email domain names Membership Organization Wiki Page  \n",
+       "0               3gpp.org                     NaN       NaN  \n",
+       "1                    NaN                    ETSI       NaN  \n",
+       "2                    NaN                   OTHER       NaN  \n",
+       "3                    NaN                   OTHER       NaN  \n",
+       "4                    NaN                   OTHER       NaN  \n",
+       "..                   ...                     ...       ...  \n",
+       "938                  NaN                    CCSA       NaN  \n",
+       "939                  NaN                    ETSI       NaN  \n",
+       "940                  NaN                    CCSA       NaN  \n",
+       "941  zte.com.cn, zte.com                    CCSA       NaN  \n",
+       "945                  NaN                    CCSA       NaN  \n",
        "\n",
-       "[666 rows x 8 columns]"
+       "[682 rows x 8 columns]"
       ]
      },
-     "execution_count": 25,
+     "execution_count": 27,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -1209,7 +1257,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 26,
+   "execution_count": 28,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -1223,7 +1271,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 27,
+   "execution_count": 29,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -1240,7 +1288,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 28,
+   "execution_count": 30,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -1250,7 +1298,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 29,
+   "execution_count": 31,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -1259,7 +1307,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 30,
+   "execution_count": 32,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -1268,7 +1316,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 31,
+   "execution_count": 33,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -1277,7 +1325,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 32,
+   "execution_count": 34,
    "metadata": {},
    "outputs": [
     {
@@ -1315,20 +1363,20 @@
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
-       "      <th>23</th>\n",
+       "      <th>24</th>\n",
        "      <td>Afilias</td>\n",
        "      <td>Internet Registry</td>\n",
        "      <td>NaN</td>\n",
        "      <td>Business</td>\n",
        "      <td>United States</td>\n",
-       "      <td>NaN</td>\n",
+       "      <td>afilias.info</td>\n",
        "      <td>NaN</td>\n",
        "      <td>NaN</td>\n",
        "      <td>Afilias</td>\n",
        "      <td>afilias.info</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>26</th>\n",
+       "      <th>27</th>\n",
        "      <td>Akamai Technologies</td>\n",
        "      <td>Content Distribution Network</td>\n",
        "      <td>NaN</td>\n",
@@ -1341,7 +1389,7 @@
        "      <td>akamai.com</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>29</th>\n",
+       "      <th>30</th>\n",
        "      <td>Alibaba (China) Group.  Ltd.</td>\n",
        "      <td>NaN</td>\n",
        "      <td>NaN</td>\n",
@@ -1354,46 +1402,46 @@
        "      <td>alibaba-inc.com</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>36</th>\n",
+       "      <th>37</th>\n",
        "      <td>Amazon Web Services  Inc.</td>\n",
        "      <td>Cloud Provider</td>\n",
        "      <td>NaN</td>\n",
        "      <td>Business</td>\n",
        "      <td>United States</td>\n",
-       "      <td>NaN</td>\n",
+       "      <td>amazon.com</td>\n",
        "      <td>ETSI</td>\n",
        "      <td>NaN</td>\n",
        "      <td>Amazon Web Services</td>\n",
        "      <td>amazon.com</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>38</th>\n",
+       "      <th>39</th>\n",
        "      <td>AMS</td>\n",
        "      <td>IETF secretariat</td>\n",
        "      <td>NaN</td>\n",
-       "      <td>Governance body</td>\n",
+       "      <td>Technical Community</td>\n",
        "      <td>United States</td>\n",
-       "      <td>NaN</td>\n",
+       "      <td>amsl.com</td>\n",
        "      <td>NaN</td>\n",
        "      <td>NaN</td>\n",
        "      <td>AMS</td>\n",
        "      <td>amsl.com</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>44</th>\n",
+       "      <th>45</th>\n",
        "      <td>APNIC</td>\n",
        "      <td>Internet Registry</td>\n",
        "      <td>NaN</td>\n",
-       "      <td>Governance body</td>\n",
-       "      <td>NaN</td>\n",
+       "      <td>Technical Community</td>\n",
        "      <td>NaN</td>\n",
+       "      <td>apnic.net</td>\n",
        "      <td>NaN</td>\n",
        "      <td>NaN</td>\n",
        "      <td>APNIC</td>\n",
        "      <td>apnic.net</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>45</th>\n",
+       "      <th>46</th>\n",
        "      <td>Apple</td>\n",
        "      <td>Consumer hardware and software vendor</td>\n",
        "      <td>NaN</td>\n",
@@ -1406,7 +1454,7 @@
        "      <td>apple.com</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>62</th>\n",
+       "      <th>63</th>\n",
        "      <td>Arista Networks</td>\n",
        "      <td>Cloud Provider</td>\n",
        "      <td>NaN</td>\n",
@@ -1419,7 +1467,7 @@
        "      <td>arista.com</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>63</th>\n",
+       "      <th>64</th>\n",
        "      <td>Arrcus</td>\n",
        "      <td>Cloud Provider</td>\n",
        "      <td>NaN</td>\n",
@@ -1432,20 +1480,20 @@
        "      <td>arrcus.com</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>64</th>\n",
-       "      <td>ARTICLE19</td>\n",
-       "      <td>Civil Society Organization</td>\n",
+       "      <th>65</th>\n",
+       "      <td>ARM</td>\n",
+       "      <td>Chipmaker</td>\n",
        "      <td>NaN</td>\n",
-       "      <td>Civil Society</td>\n",
-       "      <td>United Kingdom</td>\n",
-       "      <td>article19.org</td>\n",
-       "      <td>ETSI</td>\n",
+       "      <td>Business</td>\n",
+       "      <td>United States</td>\n",
+       "      <td>arm.com</td>\n",
        "      <td>NaN</td>\n",
-       "      <td>Article19</td>\n",
-       "      <td>article19.org</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>ARM</td>\n",
+       "      <td>arm.com</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>73</th>\n",
+       "      <th>75</th>\n",
        "      <td>AT&amp;T</td>\n",
        "      <td>Telecommunications Provider</td>\n",
        "      <td>NaN</td>\n",
@@ -1458,33 +1506,46 @@
        "      <td>att.com</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>92</th>\n",
+       "      <th>83</th>\n",
+       "      <td>Avaya</td>\n",
+       "      <td>Networking equipment vendor</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>Business</td>\n",
+       "      <td>United States</td>\n",
+       "      <td>avaya.com</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>Avaya</td>\n",
+       "      <td>avaya.com</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>95</th>\n",
        "      <td>Beijing University of Posts and Telecommunicat...</td>\n",
        "      <td>Research Institution</td>\n",
        "      <td>NaN</td>\n",
        "      <td>Academia</td>\n",
        "      <td>China</td>\n",
-       "      <td>NaN</td>\n",
+       "      <td>bupt.edu.cn</td>\n",
        "      <td>CCSA</td>\n",
        "      <td>NaN</td>\n",
        "      <td>Beijing University of Posts and Telecommunicat...</td>\n",
        "      <td>bupt.edu.cn</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>107</th>\n",
+       "      <th>110</th>\n",
        "      <td>BRITISH BROADCASTING CORPORATION</td>\n",
        "      <td>Content Provider</td>\n",
        "      <td>NaN</td>\n",
        "      <td>Civil Society</td>\n",
        "      <td>United Kingdom</td>\n",
-       "      <td>NaN</td>\n",
+       "      <td>bbc.co.uk</td>\n",
        "      <td>ETSI</td>\n",
        "      <td>NaN</td>\n",
        "      <td>British Broadcasting Corporation</td>\n",
        "      <td>bbc.co.uk</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>110</th>\n",
+       "      <th>113</th>\n",
        "      <td>BROADCOM CORPORATION</td>\n",
        "      <td>Hardware developer</td>\n",
        "      <td>NaN</td>\n",
@@ -1497,20 +1558,20 @@
        "      <td>broadcom.com</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>113</th>\n",
+       "      <th>116</th>\n",
        "      <td>BT</td>\n",
        "      <td>Telecommunications Provider</td>\n",
        "      <td>NaN</td>\n",
        "      <td>Business</td>\n",
        "      <td>United Kingdom</td>\n",
-       "      <td>bt.com</td>\n",
+       "      <td>bt.com, bellatlantic.net</td>\n",
        "      <td>ETSI</td>\n",
        "      <td>NaN</td>\n",
        "      <td>BT</td>\n",
        "      <td>bellatlantic.net</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>121</th>\n",
+       "      <th>124</th>\n",
        "      <td>CableLabs</td>\n",
        "      <td>Research Institution</td>\n",
        "      <td>NaN</td>\n",
@@ -1523,56 +1584,43 @@
        "      <td>cablelabs.com</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>123</th>\n",
+       "      <th>126</th>\n",
        "      <td>CAICT</td>\n",
        "      <td>Research Institution</td>\n",
        "      <td>NaN</td>\n",
        "      <td>Academia</td>\n",
        "      <td>China</td>\n",
-       "      <td>NaN</td>\n",
+       "      <td>caict.ac.cn</td>\n",
        "      <td>CCSA</td>\n",
        "      <td>NaN</td>\n",
        "      <td>CAICT</td>\n",
        "      <td>caict.ac.cn</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>126</th>\n",
+       "      <th>129</th>\n",
        "      <td>CAMBRIDGE CONSULTANTS LTD</td>\n",
        "      <td>Consulting</td>\n",
        "      <td>NaN</td>\n",
        "      <td>Business</td>\n",
        "      <td>United Kingdom</td>\n",
-       "      <td>NaN</td>\n",
+       "      <td>IEEE.ORG</td>\n",
        "      <td>ETSI</td>\n",
        "      <td>NaN</td>\n",
        "      <td>Consultant</td>\n",
        "      <td>IEEE.ORG</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>144</th>\n",
-       "      <td>China Academy of Information and Communication...</td>\n",
+       "      <th>132</th>\n",
+       "      <td>Carnegie Mellon University</td>\n",
        "      <td>Research Institution</td>\n",
        "      <td>NaN</td>\n",
        "      <td>Academia</td>\n",
-       "      <td>China</td>\n",
-       "      <td>NaN</td>\n",
-       "      <td>CCSA</td>\n",
-       "      <td>NaN</td>\n",
-       "      <td>China Academy of Information and Communication...</td>\n",
-       "      <td>caict.ac.cn</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <th>159</th>\n",
-       "      <td>China Mobile International Limited</td>\n",
-       "      <td>Telecommunications Provider</td>\n",
+       "      <td>United States</td>\n",
+       "      <td>sei.cmu.edu</td>\n",
        "      <td>NaN</td>\n",
-       "      <td>Business</td>\n",
-       "      <td>China</td>\n",
-       "      <td>chinamobile.com</td>\n",
-       "      <td>CCSA</td>\n",
        "      <td>NaN</td>\n",
-       "      <td>China Mobile</td>\n",
-       "      <td>chinamobile.com</td>\n",
+       "      <td>Software Engineering Institute</td>\n",
+       "      <td>sei.cmu.edu</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
@@ -1580,117 +1628,117 @@
       ],
       "text/plain": [
        "                                                  name  \\\n",
-       "23                                             Afilias   \n",
-       "26                                 Akamai Technologies   \n",
-       "29                        Alibaba (China) Group.  Ltd.   \n",
-       "36                           Amazon Web Services  Inc.   \n",
-       "38                                                 AMS   \n",
-       "44                                               APNIC   \n",
-       "45                                               Apple   \n",
-       "62                                     Arista Networks   \n",
-       "63                                              Arrcus   \n",
-       "64                                           ARTICLE19   \n",
-       "73                                                AT&T   \n",
-       "92   Beijing University of Posts and Telecommunicat...   \n",
-       "107                   BRITISH BROADCASTING CORPORATION   \n",
-       "110                               BROADCOM CORPORATION   \n",
-       "113                                                 BT   \n",
-       "121                                          CableLabs   \n",
-       "123                                              CAICT   \n",
-       "126                          CAMBRIDGE CONSULTANTS LTD   \n",
-       "144  China Academy of Information and Communication...   \n",
-       "159                 China Mobile International Limited   \n",
+       "24                                             Afilias   \n",
+       "27                                 Akamai Technologies   \n",
+       "30                        Alibaba (China) Group.  Ltd.   \n",
+       "37                           Amazon Web Services  Inc.   \n",
+       "39                                                 AMS   \n",
+       "45                                               APNIC   \n",
+       "46                                               Apple   \n",
+       "63                                     Arista Networks   \n",
+       "64                                              Arrcus   \n",
+       "65                                                 ARM   \n",
+       "75                                                AT&T   \n",
+       "83                                               Avaya   \n",
+       "95   Beijing University of Posts and Telecommunicat...   \n",
+       "110                   BRITISH BROADCASTING CORPORATION   \n",
+       "113                               BROADCOM CORPORATION   \n",
+       "116                                                 BT   \n",
+       "124                                          CableLabs   \n",
+       "126                                              CAICT   \n",
+       "129                          CAMBRIDGE CONSULTANTS LTD   \n",
+       "132                         Carnegie Mellon University   \n",
        "\n",
        "                                  category subsidiary of / alias of  \\\n",
-       "23                       Internet Registry                      NaN   \n",
-       "26            Content Distribution Network                      NaN   \n",
-       "29                                     NaN                      NaN   \n",
-       "36                          Cloud Provider                      NaN   \n",
-       "38                        IETF secretariat                      NaN   \n",
-       "44                       Internet Registry                      NaN   \n",
-       "45   Consumer hardware and software vendor                      NaN   \n",
-       "62                          Cloud Provider                      NaN   \n",
+       "24                       Internet Registry                      NaN   \n",
+       "27            Content Distribution Network                      NaN   \n",
+       "30                                     NaN                      NaN   \n",
+       "37                          Cloud Provider                      NaN   \n",
+       "39                        IETF secretariat                      NaN   \n",
+       "45                       Internet Registry                      NaN   \n",
+       "46   Consumer hardware and software vendor                      NaN   \n",
        "63                          Cloud Provider                      NaN   \n",
-       "64              Civil Society Organization                      NaN   \n",
-       "73             Telecommunications Provider                      NaN   \n",
-       "92                    Research Institution                      NaN   \n",
-       "107                       Content Provider                      NaN   \n",
-       "110                     Hardware developer                      NaN   \n",
-       "113            Telecommunications Provider                      NaN   \n",
-       "121                   Research Institution                      NaN   \n",
-       "123                   Research Institution                      NaN   \n",
-       "126                             Consulting                      NaN   \n",
-       "144                   Research Institution                      NaN   \n",
-       "159            Telecommunications Provider                      NaN   \n",
+       "64                          Cloud Provider                      NaN   \n",
+       "65                               Chipmaker                      NaN   \n",
+       "75             Telecommunications Provider                      NaN   \n",
+       "83             Networking equipment vendor                      NaN   \n",
+       "95                    Research Institution                      NaN   \n",
+       "110                       Content Provider                      NaN   \n",
+       "113                     Hardware developer                      NaN   \n",
+       "116            Telecommunications Provider                      NaN   \n",
+       "124                   Research Institution                      NaN   \n",
+       "126                   Research Institution                      NaN   \n",
+       "129                             Consulting                      NaN   \n",
+       "132                   Research Institution                      NaN   \n",
        "\n",
-       "    stakeholdergroup     nationality email domain names  \\\n",
-       "23          Business   United States                NaN   \n",
-       "26          Business   United States         akamai.com   \n",
-       "29          Business           China    alibaba-inc.com   \n",
-       "36          Business   United States                NaN   \n",
-       "38   Governance body   United States                NaN   \n",
-       "44   Governance body             NaN                NaN   \n",
-       "45          Business   United States          apple.com   \n",
-       "62          Business   United States         arista.com   \n",
-       "63         Business    United States         arrcus.com   \n",
-       "64     Civil Society  United Kingdom      article19.org   \n",
-       "73         Business    United States            att.com   \n",
-       "92          Academia           China                NaN   \n",
-       "107    Civil Society  United Kingdom                NaN   \n",
-       "110         Business   United States       broadcom.com   \n",
-       "113         Business  United Kingdom             bt.com   \n",
-       "121    Civil Society   United States      cablelabs.com   \n",
-       "123         Academia           China                NaN   \n",
-       "126         Business  United Kingdom                NaN   \n",
-       "144         Academia           China                NaN   \n",
-       "159         Business           China    chinamobile.com   \n",
+       "        stakeholdergroup     nationality        email domain names  \\\n",
+       "24              Business   United States              afilias.info   \n",
+       "27              Business   United States                akamai.com   \n",
+       "30              Business           China           alibaba-inc.com   \n",
+       "37              Business   United States                amazon.com   \n",
+       "39   Technical Community   United States                  amsl.com   \n",
+       "45   Technical Community             NaN                 apnic.net   \n",
+       "46              Business   United States                 apple.com   \n",
+       "63              Business   United States                arista.com   \n",
+       "64             Business    United States                arrcus.com   \n",
+       "65              Business   United States                   arm.com   \n",
+       "75             Business    United States                   att.com   \n",
+       "83              Business   United States                 avaya.com   \n",
+       "95              Academia           China               bupt.edu.cn   \n",
+       "110        Civil Society  United Kingdom                 bbc.co.uk   \n",
+       "113             Business   United States              broadcom.com   \n",
+       "116             Business  United Kingdom  bt.com, bellatlantic.net   \n",
+       "124        Civil Society   United States             cablelabs.com   \n",
+       "126             Academia           China               caict.ac.cn   \n",
+       "129             Business  United Kingdom                  IEEE.ORG   \n",
+       "132             Academia   United States               sei.cmu.edu   \n",
        "\n",
        "    Membership Organization Wiki Page  \\\n",
-       "23                      NaN       NaN   \n",
-       "26                      NaN       NaN   \n",
-       "29                     CCSA       NaN   \n",
-       "36                     ETSI       NaN   \n",
-       "38                      NaN       NaN   \n",
-       "44                      NaN       NaN   \n",
+       "24                      NaN       NaN   \n",
+       "27                      NaN       NaN   \n",
+       "30                     CCSA       NaN   \n",
+       "37                     ETSI       NaN   \n",
+       "39                      NaN       NaN   \n",
        "45                      NaN       NaN   \n",
-       "62                      NaN       NaN   \n",
+       "46                      NaN       NaN   \n",
        "63                      NaN       NaN   \n",
-       "64                     ETSI       NaN   \n",
-       "73                     ATIS       NaN   \n",
-       "92                     CCSA       NaN   \n",
-       "107                    ETSI       NaN   \n",
+       "64                      NaN       NaN   \n",
+       "65                      NaN       NaN   \n",
+       "75                     ATIS       NaN   \n",
+       "83                      NaN       NaN   \n",
+       "95                     CCSA       NaN   \n",
        "110                    ETSI       NaN   \n",
        "113                    ETSI       NaN   \n",
-       "121                    ETSI       NaN   \n",
-       "123                    CCSA       NaN   \n",
-       "126                    ETSI       NaN   \n",
-       "144                    CCSA       NaN   \n",
-       "159                    CCSA       NaN   \n",
+       "116                    ETSI       NaN   \n",
+       "124                    ETSI       NaN   \n",
+       "126                    CCSA       NaN   \n",
+       "129                    ETSI       NaN   \n",
+       "132                     NaN       NaN   \n",
        "\n",
        "                                              norm_org   max_domain_ietf  \n",
-       "23                                             Afilias      afilias.info  \n",
-       "26                                 Akamai Technologies        akamai.com  \n",
-       "29                                             Alibaba   alibaba-inc.com  \n",
-       "36                                 Amazon Web Services        amazon.com  \n",
-       "38                                                 AMS          amsl.com  \n",
-       "44                                               APNIC         apnic.net  \n",
-       "45                                               Apple         apple.com  \n",
-       "62                                     Arista Networks        arista.com  \n",
-       "63                                              Arrcus        arrcus.com  \n",
-       "64                                           Article19     article19.org  \n",
-       "73                                                AT&T           att.com  \n",
-       "92   Beijing University of Posts and Telecommunicat...       bupt.edu.cn  \n",
-       "107                   British Broadcasting Corporation         bbc.co.uk  \n",
-       "110                                           Broadcom      broadcom.com  \n",
-       "113                                                 BT  bellatlantic.net  \n",
-       "121                                          CableLabs     cablelabs.com  \n",
-       "123                                              CAICT       caict.ac.cn  \n",
-       "126                                         Consultant          IEEE.ORG  \n",
-       "144  China Academy of Information and Communication...       caict.ac.cn  \n",
-       "159                                       China Mobile   chinamobile.com  "
+       "24                                             Afilias      afilias.info  \n",
+       "27                                 Akamai Technologies        akamai.com  \n",
+       "30                                             Alibaba   alibaba-inc.com  \n",
+       "37                                 Amazon Web Services        amazon.com  \n",
+       "39                                                 AMS          amsl.com  \n",
+       "45                                               APNIC         apnic.net  \n",
+       "46                                               Apple         apple.com  \n",
+       "63                                     Arista Networks        arista.com  \n",
+       "64                                              Arrcus        arrcus.com  \n",
+       "65                                                 ARM           arm.com  \n",
+       "75                                                AT&T           att.com  \n",
+       "83                                               Avaya         avaya.com  \n",
+       "95   Beijing University of Posts and Telecommunicat...       bupt.edu.cn  \n",
+       "110                   British Broadcasting Corporation         bbc.co.uk  \n",
+       "113                                           Broadcom      broadcom.com  \n",
+       "116                                                 BT  bellatlantic.net  \n",
+       "124                                          CableLabs     cablelabs.com  \n",
+       "126                                              CAICT       caict.ac.cn  \n",
+       "129                                         Consultant          IEEE.ORG  \n",
+       "132                     Software Engineering Institute       sei.cmu.edu  "
       ]
      },
-     "execution_count": 32,
+     "execution_count": 34,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -1698,6 +1746,730 @@
    "source": [
     "org_cats_plus_match[:20]"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Export the graph of relations\n",
+    "\n",
+    "Getting the affiliation data relations extracted from the attendance tables.\n",
+    "\n",
+    "Final form: Three tables:\n",
+    " - Name - Email, earliest and latest date\n",
+    " - Name - Affiliation, earliest and latest date\n",
+    " - Email - Affiliation, earliest and latest date\n",
+    "\n",
+    "These can be combined into a tripartite graph, which should have a component for each affiliation entity.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 35,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "meeting_range = [106,107,108]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 36,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/home/sb/projects/bigbang/bigbang-env/lib/python3.9/site-packages/pandas/core/indexing.py:1773: SettingWithCopyWarning: \n",
+      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
+      "Try using .loc[row_indexer,col_indexer] = value instead\n",
+      "\n",
+      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
+      "  self._setitem_single_column(ilocs[0], value, pi)\n"
+     ]
+    }
+   ],
+   "source": [
+    "a, b, c = attendance.name_email_affil_relations_from_IETF_attendance(meeting_range, threshold = 0.17)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 37,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>full_name</th>\n",
+       "      <th>affiliation</th>\n",
+       "      <th>min_date</th>\n",
+       "      <th>max_date</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td></td>\n",
+       "      <td></td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>ABDEL RAHMAN NJI PANSO MOUNCHILI</td>\n",
+       "      <td></td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>AKIRA MURAKAMI</td>\n",
+       "      <td>Toshiba</td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>ALBERT ESPINAL</td>\n",
+       "      <td>ESPOL</td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>ANDREW GROVER</td>\n",
+       "      <td>Mozilla</td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>...</th>\n",
+       "      <td>...</td>\n",
+       "      <td>...</td>\n",
+       "      <td>...</td>\n",
+       "      <td>...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2674</th>\n",
+       "      <td>yusuke kagiwada</td>\n",
+       "      <td></td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2675</th>\n",
+       "      <td>zhenbin Li</td>\n",
+       "      <td></td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2676</th>\n",
+       "      <td>Éric Vyncke</td>\n",
+       "      <td>Cisco</td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2677</th>\n",
+       "      <td>Øyvind Rønningstad</td>\n",
+       "      <td>Nordic Semiconductor</td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2678</th>\n",
+       "      <td>承勳 蔡</td>\n",
+       "      <td>恩典之前  讚美之裏</td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "<p>2679 rows × 4 columns</p>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "                             full_name           affiliation   min_date  \\\n",
+       "0                                                            2019-11-16   \n",
+       "1     ABDEL RAHMAN NJI PANSO MOUNCHILI                       2019-11-16   \n",
+       "2                       AKIRA MURAKAMI               Toshiba 2019-11-16   \n",
+       "3                       ALBERT ESPINAL                 ESPOL 2019-11-16   \n",
+       "4                        ANDREW GROVER               Mozilla 2019-11-16   \n",
+       "...                                ...                   ...        ...   \n",
+       "2674                   yusuke kagiwada                       2019-11-16   \n",
+       "2675                        zhenbin Li                       2019-11-16   \n",
+       "2676                       Éric Vyncke                 Cisco 2019-11-16   \n",
+       "2677                Øyvind Rønningstad  Nordic Semiconductor 2019-11-16   \n",
+       "2678                              承勳 蔡            恩典之前  讚美之裏 2019-11-16   \n",
+       "\n",
+       "       max_date  \n",
+       "0    2020-07-25  \n",
+       "1    2020-07-25  \n",
+       "2    2020-07-25  \n",
+       "3    2020-07-25  \n",
+       "4    2020-07-25  \n",
+       "...         ...  \n",
+       "2674 2020-07-25  \n",
+       "2675 2020-07-25  \n",
+       "2676 2020-07-25  \n",
+       "2677 2020-07-25  \n",
+       "2678 2020-07-25  \n",
+       "\n",
+       "[2679 rows x 4 columns]"
+      ]
+     },
+     "execution_count": 37,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "a"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 38,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>email</th>\n",
+       "      <th>affiliation</th>\n",
+       "      <th>min_date</th>\n",
+       "      <th>max_date</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td></td>\n",
+       "      <td></td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>0bz3884c07731v.ietf@gmail.com</td>\n",
+       "      <td>SECOM</td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>115049098@qq.com</td>\n",
+       "      <td></td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>1252363@tongji.edu.cn</td>\n",
+       "      <td>Tongji University</td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>13910628561@139.com</td>\n",
+       "      <td></td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>...</th>\n",
+       "      <td>...</td>\n",
+       "      <td>...</td>\n",
+       "      <td>...</td>\n",
+       "      <td>...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2723</th>\n",
+       "      <td>zuni.wang@huawei.com</td>\n",
+       "      <td></td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2724</th>\n",
+       "      <td>zzhang1@futurewei.com</td>\n",
+       "      <td>Futurewei</td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2725</th>\n",
+       "      <td>zzhang@juniper.net</td>\n",
+       "      <td>Juniper Networks</td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2726</th>\n",
+       "      <td>zzhang_ietf@hotmail.com</td>\n",
+       "      <td></td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2727</th>\n",
+       "      <td>zzzz40820@gmail.com</td>\n",
+       "      <td>ابراهيم</td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "<p>2728 rows × 4 columns</p>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "                              email        affiliation   min_date   max_date\n",
+       "0                                                      2019-11-16 2020-07-25\n",
+       "1     0bz3884c07731v.ietf@gmail.com              SECOM 2019-11-16 2020-07-25\n",
+       "2                  115049098@qq.com                    2019-11-16 2020-07-25\n",
+       "3             1252363@tongji.edu.cn  Tongji University 2019-11-16 2020-07-25\n",
+       "4               13910628561@139.com                    2019-11-16 2020-07-25\n",
+       "...                             ...                ...        ...        ...\n",
+       "2723           zuni.wang@huawei.com                    2019-11-16 2020-07-25\n",
+       "2724          zzhang1@futurewei.com          Futurewei 2019-11-16 2020-07-25\n",
+       "2725             zzhang@juniper.net   Juniper Networks 2019-11-16 2020-07-25\n",
+       "2726        zzhang_ietf@hotmail.com                    2019-11-16 2020-07-25\n",
+       "2727            zzzz40820@gmail.com            ابراهيم 2019-11-16 2020-07-25\n",
+       "\n",
+       "[2728 rows x 4 columns]"
+      ]
+     },
+     "execution_count": 38,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "b"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 39,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "5"
+      ]
+     },
+     "execution_count": 39,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "b['affiliation'].value_counts()['cisco']"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 40,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>full_name</th>\n",
+       "      <th>email</th>\n",
+       "      <th>min_date</th>\n",
+       "      <th>max_date</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td></td>\n",
+       "      <td>I</td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-03-21</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td></td>\n",
+       "      <td>haqoleho@mailinator.com</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>ABDEL RAHMAN NJI PANSO MOUNCHILI</td>\n",
+       "      <td>morellaorny@gmail.com</td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>AKIRA MURAKAMI</td>\n",
+       "      <td>akira4.murakami@toshiba.co.jp</td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>ALBERT ESPINAL</td>\n",
+       "      <td>aespinal@espol.edu.ec</td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>...</th>\n",
+       "      <td>...</td>\n",
+       "      <td>...</td>\n",
+       "      <td>...</td>\n",
+       "      <td>...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2587</th>\n",
+       "      <td>yusuke kagiwada</td>\n",
+       "      <td>block.rxckin.beats@gmail.com</td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2588</th>\n",
+       "      <td>zhenbin Li</td>\n",
+       "      <td>robinli314@163.com</td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2589</th>\n",
+       "      <td>Éric Vyncke</td>\n",
+       "      <td>evyncke@cisco.com</td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2590</th>\n",
+       "      <td>Øyvind Rønningstad</td>\n",
+       "      <td>oyvind.ronningstad@nordicsemi.no</td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2591</th>\n",
+       "      <td>承勳 蔡</td>\n",
+       "      <td>caiyisho@gmail.com</td>\n",
+       "      <td>2019-11-16</td>\n",
+       "      <td>2020-07-25</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "<p>2592 rows × 4 columns</p>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "                             full_name                             email  \\\n",
+       "0                                                                      I   \n",
+       "1                                                haqoleho@mailinator.com   \n",
+       "2     ABDEL RAHMAN NJI PANSO MOUNCHILI             morellaorny@gmail.com   \n",
+       "3                       AKIRA MURAKAMI     akira4.murakami@toshiba.co.jp   \n",
+       "4                       ALBERT ESPINAL             aespinal@espol.edu.ec   \n",
+       "...                                ...                               ...   \n",
+       "2587                   yusuke kagiwada      block.rxckin.beats@gmail.com   \n",
+       "2588                        zhenbin Li                robinli314@163.com   \n",
+       "2589                       Éric Vyncke                 evyncke@cisco.com   \n",
+       "2590                Øyvind Rønningstad  oyvind.ronningstad@nordicsemi.no   \n",
+       "2591                              承勳 蔡                caiyisho@gmail.com   \n",
+       "\n",
+       "       min_date   max_date  \n",
+       "0    2019-11-16 2020-03-21  \n",
+       "1    2020-07-25 2020-07-25  \n",
+       "2    2019-11-16 2020-07-25  \n",
+       "3    2019-11-16 2020-07-25  \n",
+       "4    2019-11-16 2020-07-25  \n",
+       "...         ...        ...  \n",
+       "2587 2019-11-16 2020-07-25  \n",
+       "2588 2019-11-16 2020-07-25  \n",
+       "2589 2019-11-16 2020-07-25  \n",
+       "2590 2019-11-16 2020-07-25  \n",
+       "2591 2019-11-16 2020-07-25  \n",
+       "\n",
+       "[2592 rows x 4 columns]"
+      ]
+     },
+     "execution_count": 40,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "c"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Match to a mailing list"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 41,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from bigbang.archive import Archive\n",
+    "arx = Archive(\"httpbisa\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "From the archive data: From -> email address, Date\n",
+    "\n",
+    "Match with table B: email,. min_date, max_date, to get Affiliation\n",
+    "\n",
+    "Add Affiliation to the archive data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 42,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "arx.add_affiliation(b)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 43,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>From</th>\n",
+       "      <th>Date</th>\n",
+       "      <th>affiliation</th>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>Message-ID</th>\n",
+       "      <th></th>\n",
+       "      <th></th>\n",
+       "      <th></th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>&lt;DM6PR22MB20103E911E1FE1007270CEA9DA4D0@DM6PR22MB2010.namprd22.prod.outlook.com&gt;</th>\n",
+       "      <td>Mike Bishop &lt;mbishop@evequefou.be&gt;</td>\n",
+       "      <td>2019-11-18 08:02:15+00:00</td>\n",
+       "      <td>Akamai Technologies</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>&lt;CALGR9obJLrTufZe+UGtzNmucyZa4oQCcOeuY+Aq9SwM4w0kaJg@mail.gmail.com&gt;</th>\n",
+       "      <td>Lucas Pardue &lt;lucaspardue.24.7@gmail.com&gt;</td>\n",
+       "      <td>2019-11-18 18:55:17+00:00</td>\n",
+       "      <td>Cloudflare</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>&lt;C573DE86-39B5-48AB-9C21-0717F4EBEB59@mnot.net&gt;</th>\n",
+       "      <td>Mark Nottingham &lt;mnot@mnot.net&gt;</td>\n",
+       "      <td>2019-11-19 07:43:15+00:00</td>\n",
+       "      <td>Fastly</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>&lt;CAKC-DJiuE0Gmt5kMz8UJbZ625hdeMZLih3CCg1FQKx92D79oLw@mail.gmail.com&gt;</th>\n",
+       "      <td>Erik Nygren &lt;erik+ietf@nygren.org&gt;</td>\n",
+       "      <td>2019-11-20 02:55:26+00:00</td>\n",
+       "      <td>Akamai Technologies</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>&lt;371380E9-7204-41EC-8F32-653E9B5272D8@iii.ca&gt;</th>\n",
+       "      <td>Cullen Jennings &lt;fluffy@iii.ca&gt;</td>\n",
+       "      <td>2019-11-20 09:39:43+00:00</td>\n",
+       "      <td>cisco</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>...</th>\n",
+       "      <td>...</td>\n",
+       "      <td>...</td>\n",
+       "      <td>...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>&lt;CAAZdMae-jZVfv98jC3ru51uk6H6TZuaQ+P=PJW-BZ9uEKVbQCA@mail.gmail.com&gt;</th>\n",
+       "      <td>Victor Vasiliev &lt;vasilvv@google.com&gt;</td>\n",
+       "      <td>2020-07-21 20:22:30+00:00</td>\n",
+       "      <td>Google</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>&lt;CALGR9oa1aW90VmrGO8wAsBWAq2un5bHY67m-HpaR_8LpNrD_3g@mail.gmail.com&gt;</th>\n",
+       "      <td>Lucas Pardue &lt;lucaspardue.24.7@gmail.com&gt;</td>\n",
+       "      <td>2020-07-21 23:43:46+00:00</td>\n",
+       "      <td>Cloudflare</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>&lt;010001737c434b23-44eb8c59-f98a-4c3d-8cf3-b991994f2e23-000000@email.amazonses.com&gt;</th>\n",
+       "      <td>Kent Watsen &lt;kent+ietf@watsen.net&gt;</td>\n",
+       "      <td>2020-07-23 15:20:51+00:00</td>\n",
+       "      <td>Watsen Networks</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>&lt;CAHbrMsDNwZN64Y7Tfp0e0JQOSfArk5LeUTC8JqBeatiBVFJN0g@mail.gmail.com&gt;</th>\n",
+       "      <td>Ben Schwartz &lt;bemasc@google.com&gt;</td>\n",
+       "      <td>2020-07-23 18:06:21+00:00</td>\n",
+       "      <td>Google / Jigsaw</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>&lt;010001737d8fd2a0-48a56328-5a22-4a27-afb7-1729d16e0fe5-000000@email.amazonses.com&gt;</th>\n",
+       "      <td>Kent Watsen &lt;kent+ietf@watsen.net&gt;</td>\n",
+       "      <td>2020-07-23 21:24:04+00:00</td>\n",
+       "      <td>Watsen Networks</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "<p>320 rows × 3 columns</p>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "                                                                                         From  \\\n",
+       "Message-ID                                                                                      \n",
+       "<DM6PR22MB20103E911E1FE1007270CEA9DA4D0@DM6PR22...         Mike Bishop <mbishop@evequefou.be>   \n",
+       "<CALGR9obJLrTufZe+UGtzNmucyZa4oQCcOeuY+Aq9SwM4w...  Lucas Pardue <lucaspardue.24.7@gmail.com>   \n",
+       "<C573DE86-39B5-48AB-9C21-0717F4EBEB59@mnot.net>               Mark Nottingham <mnot@mnot.net>   \n",
+       "<CAKC-DJiuE0Gmt5kMz8UJbZ625hdeMZLih3CCg1FQKx92D...         Erik Nygren <erik+ietf@nygren.org>   \n",
+       "<371380E9-7204-41EC-8F32-653E9B5272D8@iii.ca>                 Cullen Jennings <fluffy@iii.ca>   \n",
+       "...                                                                                       ...   \n",
+       "<CAAZdMae-jZVfv98jC3ru51uk6H6TZuaQ+P=PJW-BZ9uEK...       Victor Vasiliev <vasilvv@google.com>   \n",
+       "<CALGR9oa1aW90VmrGO8wAsBWAq2un5bHY67m-HpaR_8LpN...  Lucas Pardue <lucaspardue.24.7@gmail.com>   \n",
+       "<010001737c434b23-44eb8c59-f98a-4c3d-8cf3-b9919...         Kent Watsen <kent+ietf@watsen.net>   \n",
+       "<CAHbrMsDNwZN64Y7Tfp0e0JQOSfArk5LeUTC8JqBeatiBV...           Ben Schwartz <bemasc@google.com>   \n",
+       "<010001737d8fd2a0-48a56328-5a22-4a27-afb7-1729d...         Kent Watsen <kent+ietf@watsen.net>   \n",
+       "\n",
+       "                                                                        Date  \\\n",
+       "Message-ID                                                                     \n",
+       "<DM6PR22MB20103E911E1FE1007270CEA9DA4D0@DM6PR22... 2019-11-18 08:02:15+00:00   \n",
+       "<CALGR9obJLrTufZe+UGtzNmucyZa4oQCcOeuY+Aq9SwM4w... 2019-11-18 18:55:17+00:00   \n",
+       "<C573DE86-39B5-48AB-9C21-0717F4EBEB59@mnot.net>    2019-11-19 07:43:15+00:00   \n",
+       "<CAKC-DJiuE0Gmt5kMz8UJbZ625hdeMZLih3CCg1FQKx92D... 2019-11-20 02:55:26+00:00   \n",
+       "<371380E9-7204-41EC-8F32-653E9B5272D8@iii.ca>      2019-11-20 09:39:43+00:00   \n",
+       "...                                                                      ...   \n",
+       "<CAAZdMae-jZVfv98jC3ru51uk6H6TZuaQ+P=PJW-BZ9uEK... 2020-07-21 20:22:30+00:00   \n",
+       "<CALGR9oa1aW90VmrGO8wAsBWAq2un5bHY67m-HpaR_8LpN... 2020-07-21 23:43:46+00:00   \n",
+       "<010001737c434b23-44eb8c59-f98a-4c3d-8cf3-b9919... 2020-07-23 15:20:51+00:00   \n",
+       "<CAHbrMsDNwZN64Y7Tfp0e0JQOSfArk5LeUTC8JqBeatiBV... 2020-07-23 18:06:21+00:00   \n",
+       "<010001737d8fd2a0-48a56328-5a22-4a27-afb7-1729d... 2020-07-23 21:24:04+00:00   \n",
+       "\n",
+       "                                                            affiliation  \n",
+       "Message-ID                                                               \n",
+       "<DM6PR22MB20103E911E1FE1007270CEA9DA4D0@DM6PR22...  Akamai Technologies  \n",
+       "<CALGR9obJLrTufZe+UGtzNmucyZa4oQCcOeuY+Aq9SwM4w...           Cloudflare  \n",
+       "<C573DE86-39B5-48AB-9C21-0717F4EBEB59@mnot.net>                  Fastly  \n",
+       "<CAKC-DJiuE0Gmt5kMz8UJbZ625hdeMZLih3CCg1FQKx92D...  Akamai Technologies  \n",
+       "<371380E9-7204-41EC-8F32-653E9B5272D8@iii.ca>                     cisco  \n",
+       "...                                                                 ...  \n",
+       "<CAAZdMae-jZVfv98jC3ru51uk6H6TZuaQ+P=PJW-BZ9uEK...               Google  \n",
+       "<CALGR9oa1aW90VmrGO8wAsBWAq2un5bHY67m-HpaR_8LpN...           Cloudflare  \n",
+       "<010001737c434b23-44eb8c59-f98a-4c3d-8cf3-b9919...      Watsen Networks  \n",
+       "<CAHbrMsDNwZN64Y7Tfp0e0JQOSfArk5LeUTC8JqBeatiBV...      Google / Jigsaw  \n",
+       "<010001737d8fd2a0-48a56328-5a22-4a27-afb7-1729d...      Watsen Networks  \n",
+       "\n",
+       "[320 rows x 3 columns]"
+      ]
+     },
+     "execution_count": 43,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "arx.data[['From','Date','affiliation']].dropna()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {
@@ -1716,7 +2488,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.5"
+   "version": "3.9.7"
   },
   "latex_envs": {
    "LaTeX_envs_menu_present": true,


Test patch:
diff --git a/tests/unit/test_archive.py b/tests/unit/test_archive.py
index b11a75be..e030946f 100644
--- a/tests/unit/test_archive.py
+++ b/tests/unit/test_archive.py
@@ -1,16 +1,10 @@
-import email
-import logging
-import mailbox
+import dateutil
 import os
-import pathlib
-import re
-import shutil
 from pathlib import Path
+import shutil
 import unittest
 
-import networkx as nx
 import pandas as pd
-from testfixtures import LogCapture
 
 import bigbang.archive as archive
 from config.config import CONFIG
@@ -64,3 +58,31 @@ def test__open_list_archives(self):
         )
         assert len(data.columns.values) == 6
         assert len(data.index.values) == 108
+
+        ## Testing add_affilation
+
+        rel_email_affil = pd.DataFrame.from_records(
+            [
+                {
+                    "email": "wangruixin@caict.ac.cn",
+                    "affiliation": "TestOrg",
+                    "min_date": dateutil.parser.parse("2019-04-18"),
+                    "max_date": dateutil.parser.parse("2021-04-18"),
+                }
+            ]
+        )
+
+        arx = archive.Archive(data)
+
+        arx.add_affiliation(rel_email_affil)
+
+        assert (
+            arx.data.loc["028c01d4fa43$13f2d060$3bd87120$@caict.ac.cn"][
+                "affiliation"
+            ]
+            == "TestOrg"
+        )
+        assert (
+            arx.data.loc["2019030112374440934718@caict.ac.cn"]["affiliation"]
+            is None
+        )