Repository: data-for-change/anyway

Issue:
Update united.py to load from new CSV format
The CSV files received from United Hatzalah now has another column: the first column is the Mission ID, which will be used as the `id` field instead of the convoluted solution we used with the data and line number.
Once the script is updated, we'll remove all existing United accidents from the DB (due to the change in the `id` field) and re-load everything.

<!---
@huboard:{"order":8.113787043839693e-05,"milestone_order":0.14453125,"custom_state":""}
-->



Discussion:
Related to #427

@danielhers - Should we ignore all past accidents with the old csv structure?

Yes. We can always ask for a historical report with all the required columns.

So it would be also okay to delete all gmail data with old structure too...

Right.

On Fri, Oct 9, 2015 at 12:54 PM Omer Hamerman notifications@github.com
wrote:

> So it would be also okay to delete all gmail data with old structure too...
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/hasadna/anyway/issues/457#issuecomment-146817622.

Do you have exact date and time point of change from which I should keep?

October 6, 2015, 10:00AM

On Fri, Oct 9, 2015 at 1:02 PM Omer Hamerman notifications@github.com
wrote:

> Do you have exact date and time point of change from which I should keep?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/hasadna/anyway/issues/457#issuecomment-146819994.


Resolving patch:
diff --git a/importmail.py b/importmail.py
--- a/importmail.py
+++ b/importmail.py
@@ -76,6 +76,11 @@ def main(username=None, password=None, lastmail=False):
             else:
                 continue
 
+        # Handles Gmail bug which hasn't physically removed some of the deleted files
+        mail_date = datetime(2015, 10, 06, 10)
+        if mtime < mail_date:
+            continue
+
         for part in mail.walk():
             if part.get_content_maintype() == 'multipart' or part.get('Content-Disposition') is None:
                 continue
diff --git a/united.py b/united.py
--- a/united.py
+++ b/united.py
@@ -38,8 +38,7 @@ def create_accidents(file_location):
     :return: Yields a marker object with every iteration
     """
     print("\tReading accidents data from '%s'..." % file_location)
-    inc = 1
-    csvmap = {"time": 0, "lat": 1, "long": 2, "street": 3, "city": 5, "comment": 6, "type": 7, "casualties": 8}
+    csvmap = {"id": 0, "time": 1, "lat": 2, "long": 3, "street": 4, "city": 6, "comment": 7, "type": 8, "casualties": 9}
 
     with open(file_location, 'rU') as f:
         reader = csv.reader(f, delimiter=',', dialect=csv.excel_tab)
@@ -56,11 +55,11 @@ def create_accidents(file_location):
 
             created = parse_date(accident[csvmap["time"]])
             marker = {
+                "id": accident[csvmap["id"]],
                 "latitude": accident[csvmap["lat"]],
                 "longitude": accident[csvmap["long"]],
                 "created": created,
                 "provider_code": PROVIDER_CODE,
-                "id": int(created.strftime("%s")[2:-1] + "%02d" % inc),  # must be < 2147483647
                 "title": unicode(accident[csvmap["type"]], encoding='utf-8'),
                 "address": unicode((accident[csvmap["street"]] + ' ' + accident[csvmap["city"]]), encoding='utf-8'),
                 "severity": 2 if u"קשה" in unicode(accident[csvmap["type"]], encoding='utf-8') else 3,
@@ -71,10 +70,6 @@ def create_accidents(file_location):
                 "description": unicode(accident[csvmap["comment"]], encoding='utf-8')
             }
 
-            inc += 1
-            if inc >= 100:  # id will be too big for PostgreSQL
-                print "\t\tToo many accidents in file..."
-                break
             yield marker
 
 


Test patch: