From: beaurega@ireq.hydro.qc.ca (Denis Beauregard)
Subject: Merging GEDCOM files
Summary: What is the best strategy?
Organization: Institut de recherche d'Hydro-Quebec, Varennes, Canada
Date: Mon, 21 Dec 1992 15:42:31 GMT

Afraid of TECHNI-speach?  Depress N for Next now!

What is the best strategy to merge 2 GEDCOM files and eliminate duplicates?

Here is the one I would like to use for a GEDCOM merger program.


1- Sort data by date.  Try to have data in order of birth year.
If no birth year, invent one (i.e. date of marriage minus 20/25 years,
depending on woman/man, check before for a realistic mean age at marriage).
Don't forget to separate date so as to be able to discard it after sorting!

2- Compare couples.  Merge if names are the same (4 on 4) and years
are compatible (i.e. respect birth/wed/death years sequence).

3- Make a good comparison function.  For example, accept SOUND similar
names, i.e. accept BEAUREGARD = BOURGARD, reject BEAUREGARD = BERGER,
accept JONES = JOHNS, reject JOHNS = JOHNSON
Accept 4 names on 4 as in 2.

4- Build a comparison table.  That is, a table of aliases.  For example,
such a table could include entries like:

1 BEAUREGARD BORGARD    1= perfect phonetic match
1 BEAUREGARD BEARGARD
2 JARRET CHAREST        2= not perfect phonetic match
2 JARRET SAURET
2 JARRET SOREL
3 BEAUREGARD JARRET     3= dit name/surname match (for French roots)
4 BEAUREGARD BOGART     4= rejected match or questionable match

To build that table, the program could use a rough comparison function
i.e. accept BEAUREGARD = BERGER, JOHNS = JOHNSON

Just produce the table from data you have and delete illegal forms by hand.

5- Redo step 3, using this time the comparison table built in 3.

6- Redo step 2 to 5, and this time check for more checks:
3 names on 4 (i.e. one different first name or family name)
2 family names, same marriage date and place



In steps 3-6, accept equivalent first names.  i.e. Elizabeth = Elisabeth
Count 4 points for exact match, 2 points (or variable) for partial match
(i.e. Elisa :: Elisabeth is 3 points, John :: Jonathan is 1 point).

You could use the same points system for family name.  A perfect match
is 4 points, a not perfect is about 2, etc.

I would give 
4 points for a perfect match
    (BEAUREGARD = JARRET)
3 points for a good match, usual or common, but which is not 100%,
    (BEAUREGARD = BOUREGARD)
2 points for a mean match, typical errors in retyping
    (BEAUREGARD = BOURGARD, because BOURGARD is also another name)
1 point for a possible match, rarer errors
    (BEAUREGARD = BEAUGRAND, a mismatch I found often)
0 point for a known false match
    (to eliminate weird combination since the rough compare is not perfect)

I think, ideally, I would use the 4 points system when comparing a couple.
14-16 is a perfect match (accept on 1st pass)
11-13 is a good match (accept if more data validate the match, i.e. years)
 8-10 is a match to check by hand.
These values are not optimised at all.


Other things I did not mention:  Standardize places.  Use a place file
for that purpose.  I did so using this strategy: remove spaces from a
place name and use a standard short code.  I am currently scanning
a book with links.  My program reads the previous places file and
display new places as they are met.  

Standardize dates and other data too.  If merging GEDCOM, standardize
keywords.


To summarize, I need 3 files: first names, family names and place names.
The program can generate a file with new/unknown/presumed matches.
Also, you could use pseudo-names, like ELIZABETH -> %ELIZ.


Comments are welcome.

-- 
  \_\    Denis Beauregard  *  internet:beaurega@ireq.hydro.qc.ca
 /   \   Ge'ne'alogiste des familles : Beauregard/Jarret/Jarest/Vincent
J   __>  Un Que'bec renouvele' dans une Ame'rique renove'e
\_.-===  Ope'rateur de "Racines du Que'bec" (514) 922-9636 BBS/Genealogie
Newsgroups: soc.roots