gnowledge/gstudio — pull request #696 by AvadootNachankar (2014-08-24 12:52:44+00:00)
Title: Bug fix: Inconsistent/Invalid structure of instances of classes used for map-reducing

* Perform following:

1) python manage.py filldb

2) python manage.py sync_existing_documents

3) python manage.py map_reduce

- If it gives one of the following error or similar to this kind, it means the structure of your instances (or documents) of classes used for map-reducing is inconsistent/invalid:

1. MapReduceError: unknown fields [u'rating', u'modified_by', u'language', u'license', u'collection_set', u'post_node', u'group_set', u'property_order', u'location', u'annotations', u'access_policy'] in ReducedDocs!!!

2. MapReduceError: missed fields ['_type'] in ReducedDocs!!!

4) This only happen, if you have executed sync_existing_documents.py script after execution of scripts (map_reduce.py & search_scripts.py) required for search module.

- Reason: Few queries in sync_existing_docuemnts.py script are not excluding instances of these classes as these introduced later!

- So when you execute sync_existing_docuemnts.py script, it's appending new fields into those instances as well. This is the cause for inconsistency.

5) Just to confirm above point, goto shell and execute following commands:

- from gnowsys_ndf.ndf.models import *
- collection = get_database()[Node.collection_name]

1. rd_cur = collection.ReducedDocs.find({'is_indexed': {'$exists': True}, 'license': {'$exists': True}})
- Check what count comes (rd_cur.count()): If it's > 0, then it means you have those many inconsistent document(s) of ReducedDocs class

2. trd_cur = collection.ToReduceDocs.find({'doc_id': {'$exists': True}, 'license': {'$exists': True}})
- Check what count comes (trd_cur.count()): If it's > 0, then it means you have those many inconsistent document(s) of ToReduceDocs class

3. iwl_cur = collection.IndexedWordList.find({'word_start_id': {'$exists': True}, 'license': {'$exists': True}})
- Check what count comes (iwl_cur.count()): If it's > 0, then it means you have those many inconsistent document(s) of IndexedWordList class

6) This is resolved by doing required changes in search_scripts.py script itself which cleans unwanted fields from these instances.

- Files modified:

1) management/commands/search_scripts.py
2) models.py - ```_type field added into ReducedDocs, ToReduceDocs, IndexedWordList classes```
3) management/commands/sync_existing_documents.py
4) management/commands/map_reduce.py

7) In order to resolve this problem take this commit, and execute following command:

- **python manage.py search_scripts -c --no-reduce**