Create unique index on one column

to avoid duplicate records.
Update records, instead of insert them.
Print out date to see where we are in scraping