Repository: akvo/akvo-rsr

Issue:
IATI Import enhancements
1. [x] The code should be refactored (see https://gist.github.com/zzgvh/834b2c7b964bb57aa360)
2. [x] Add a lightweight async queue for handling scheduled imports
3. [x] Add option for not updating existing photos and improve photo handling in general



Discussion:
# Usage of the IATI import refactoring

Importing IATI data into RSR can now be set up to automatically load data regularly. This document describes the steps needed to do this and some of the technical details for running the code.

### Overview

The IATI import works with four models, **IatiImport**, **IatiImportJob**, **IatiActivityImport** and **IatiImportLog**. The IatiImport defines an import by setting the URL to the IATI file and the frequency by which it is checked. The IatiImportJob represents one running of an IatiImport, and allows the manual addition of an IATI file. The IatiActivityImport holds the details of the import of one activity in a job. Finally the IatiImportLog is the model that is used to record all that is happening during an import and it provides the data for the report email created when a job is completed. The relevant records from the log can be viewed inline in the IatiImportJob and IatiActivityImport views in the admin.

### Setting up an import

The default usage assumes that an IATI file is available via a URL that is checked regularly. When the IATI file changes an import is run automatically to reflect the changes in the file in RSR. Setting up an import is done in the RSR admin by creating a new IatiImport.

![screenshot_101](https://cloud.githubusercontent.com/assets/142613/12288466/fdc71500-b9d4-11e5-9941-69a96acbe9f4.png)
The setup above is checking once per week for the Every child IATI file hosted at aidstream.org.

The reporting-org must be present in RSR already and be allowed to create projects.

#### Notes on IatiImport fields

**Custom mappers** This field holds a drop-down list of custom mappers, i.e. code customized for a particular partner and/or IATI file format. Currently there are three, ICCO, CordaidZip and Cordaid.

ICCO is for the importing of ICCO IATI activities, and the other two are for importing Cordaid activities. The ICCO mapper is under development, I don't think ICCO are 100% done with their file format.

CordaidZip implements importing from the Zip archive we get from Cordaid every quarter. It loads both organisations and projects and works by manually uploading the whole zip file we get from Cordaid.

The Cordaid custom mapper is a sub set of CordaidZip. It is used for testing of the Coradi IATI XML only, also by manually uploading the file.

**Scheduled import** To get this working a cron job is run (currently every minute, but we might want to lower the frequency for production) that calls the iati_import managment command. To set up this cron job you need to run `manage.py crontab add`. To check that the cronjob is in place use `manage.py crontab show`. To remove the cron job use `manage.py crontab remove`.

**Run immediately**  If this box is checked the import is run "immediately", meaning the next time the cron job kicks in. This might mean a delay equal to the frequency of the cron script execution before the job is actually started.

It is also possible to run an import "fully manual" from the command line. This is useful when you want to test using an uplaoded IATI XML file. After setting up an IatiImport to run immediately, create an IatiImportJob, selecting the IatiImport you want it to be run for, and optionally uploading an IATI file. When doign this kind of testing it is recommended that the cron job is not active as this can interfere with the manual running of the import.

![screenshot_102](https://cloud.githubusercontent.com/assets/142613/12289521/5843247c-b9dc-11e5-8126-656943794c5a.png)
When the job has been set up this way, you can run `manage.py iati_import`from the command line to trigger running it directly. Note that this will trigger any other pending jobs too, so it's not recommended doing this in production!

# Test plan

Testing this is hard in some respects. The actual scheduled running should be straightforward to test. Set up an IatiImport with a short schedule and see that it runs repeatedly. After the first import job the following should do "nothing" unless the source IATI file changes. See comment above for details on setting things up.

The tricky bit is making sure the data is correctly imported. One idea (I haven't tested this) is to create a test project in RSR and populate it with **everything**. All possible data that a project can hold should have data. Where possible the data should say what data it is so for example the short_description can be "Short description text" for easy identification. Using this project, create an IATI file using the RSR IATI export functionality. Load the export using the IatiImport and check that the project is unchanged. Alternatively (probably better) is to change the IATI activity ID so that a new project is created and check that it holds data identical to the original project. In all likelihood this will reveal problems in the importer and possibly in the exporter!


Resolving patch:
diff --git a/akvo/rsr/views/my_rsr.py b/akvo/rsr/views/my_rsr.py
--- a/akvo/rsr/views/my_rsr.py
+++ b/akvo/rsr/views/my_rsr.py
@@ -307,12 +307,12 @@ def my_reports(request):
     for employment in user.approved_employments():
         org = employment.organisation
         organisations_data.append({'id': org.pk,
-                                   'name': '{0} ({1})'.format(org.name, org.long_name)})
+                                   'name': u'{0} ({1})'.format(org.name, org.long_name)})
         for project in org.all_projects():
             if project.pk not in [project_data['id'] for project_data in projects_data]:
                 projects_data.append({'id': project.pk,
-                                      'name': '{0} (id: {1})'.format(project.title,
-                                                                     str(project.pk))})
+                                      'name': u'{0} (id: {1})'.format(project.title,
+                                                                      str(project.pk))})
 
     context = {
         'reports_data': json.dumps(REPORTS),


Test patch: