Fixed project_tree, now only looking inside src/

If the project root folder contains for example a build/ or a dist/
folder, which may contain some of the sub folders which project_tree is
looking for, then the result may not bee the expected.

This error is fixed by forcing project_tree to search for sub folders
inside the src/ folder only.
Minor bug fixes of original code.

The super call in __init__ of the MaltegoException class was given the
Super class and not the sub class.

The appendelement and removeelement of the _Entity class was trying to
add and remove elements from the underlying fields and labels variables
as if they were lists. However they were actually dictionaries.
Merge pull request #32 from reenberg/fix/project_tree

Fixed project_tree, now only looking inside src/
Merge pull request #34 from reenberg/fix/maltego-message

Minor bug fixes of original code.
Fix stupid defaults in safedexml.

We are dealing with ordinarry XML so threat it that way.

It was impossible to parse the following valid XML from Maltego, without
this fix, as it required the elements to be ordered according to how
they were defined in the c.m.m._Entity class.

---------------------------------------------------------------------------

import canari.maltego.message as msg

msg._Entity.parse('''
<Entity Type="Domain">
  <AdditionalFields>
    <Field Name="fqdn" DisplayName="Domain Name">paterva.com</Field>
  </AdditionalFields>
  <Weight>66</Weight>
  <Value>paterva.com</Value>
</Entity>
''')

---------------------------------------------------------------------------
ParseError    Traceback (most recent call last)
<ipython-input-2-abb8246f3cd9> in <module>()
      7             <Value>paterva.com</Value>
      8          </Entity>
----> 9 ''')

/usr/local/lib/python2.7/dist-packages/safedexml/__init__.py in
parse(cls, xml)
    351             if field.required and field not in fields_found:
    352                 err = "required field not found: '%s'"
    % (field.field_name,)
--> 353                 raise ParseError(err)
    354             field.parse_done(self)
    355         #  All done, return the instance so created

ParseError: required field not found: 'value'
Merge branch 'fix/proper-defaults-for-safedexml' into feature/maltego-message

Conflicts:
	src/canari/maltego/message.py
Removed mandatory=False from entities in MaltegoTransformRequestMessage

When parsing the following valid XML from Maltego, the 'entities' field
is actually not parsed but silently ignored, due to the fact
that it was marked optional.

Now the following code will throw an ParseError, which needs to be
fixed. To aid this, I have added a debug version of 'safedexml.py' in
the xmltools folder, which may be copied into your path, and thus output
debug info when calling the 'parse' function. This debug output is also
included at the end.

---------------------------------------------------------------------------

In [1]: import canari.maltego.message as msg

In [2]: mmsg = msg.MaltegoMessage.parse('''
<MaltegoMessage>
   <MaltegoTransformRequestMessage>
      <Entities>
         <Entity Type="Domain">
            <AdditionalFields>
               <Field Name="fqdn" DisplayName="Domain
               Name">paterva.com</Field>
            </AdditionalFields>
            <Weight>0</Weight>
            <Value>paterva.com</Value>
         </Entity>
      </Entities>
      <Limits SoftLimit="255" HardLimit="255"/>
   </MaltegoTransformRequestMessage>
</MaltegoMessage>
''')

---------------------------------------------------------------------------

ParseError                                Traceback (most recent call
last)
<ipython-input-3-346b231349d0> in <module>()
     14    </MaltegoTransformRequestMessage>
     15 </MaltegoMessage>
---> 16 ''')

/usr/local/lib/python2.7/dist-packages/safedexml/__init__.py in
parse(cls, xml)
    343             print nest(), 'self._fields:', str(self._fields)
    344             print nest(), 'fields found:', fields_found
--> 345             self._parse_children_unordered(node, self._fields,
fields_found)
    346         #  Check that all required fields have been found
    347

/usr/local/lib/python2.7/dist-packages/safedexml/__init__.py in
_parse_children_unordered(self, node, fields, fields_found)
    402                     continue
    403                 field = fields[idx]
--> 404                 res = field.parse_child_node(self, child)
    405                 if res is PARSE_DONE:
    406                     done_fields[idx] = True

/usr/local/lib/python2.7/dist-packages/safedexml/fields.pyc in
parse_child_node(self, obj, node)
    809             field.field_name = self.field_name
    810             field.model_class = self.model_class
--> 811             res = field.parse_child_node(obj, node)
    812             if res is dexml.PARSE_MORE:
    813                 raise ValueError("items in a Choice cannot
    return PARSE_MORE")

/usr/local/lib/python2.7/dist-packages/safedexml/fields.pyc in
parse_child_node(self, obj, node)
    474             return dexml.PARSE_SKIP
    475         else:
--> 476             inst = typeclass.parse(node)
    477             self.__set__(obj, inst)
    478             return dexml.PARSE_DONE

/usr/local/lib/python2.7/dist-packages/safedexml/__init__.py in
parse(cls, xml)
    351             if field.required and field not in fields_found:
    352                 err = "required field not found: '%s'"
    % (field.field_name,)
--> 353                 raise ParseError(err)
    354             field.parse_done(self)
    355         #  All done, return the instance so created

ParseError: required field not found: 'entities'

---------------------------------------------------------------------------

For reference, I have added some debug output from safedexml, which may
help to diagnose the issue.

It is seen that when it constructs the class
MaltegoTransformRequestMessage, then it goes straight to creating the
Limits class.

---------------------------------------------------------------------------

 parse(....) ------------------------------------------------------------------------------------
        Constructing class: MaltegoMessage
        Generating node from XML: '\n<MaltegoMessage>\n   <MaltegoTransformRequestMessage>\n      <Entities>\n         <Entity Type="Domain">\n            <AdditionalFields>\n               <Field Name="fqdn" DisplayName="Domain Name">paterva.com</Field>\n            </AdditionalFields>\n            <Weight>0</Weight>\n            <Value>paterva.com</Value>\n         </Entity>\n      </Entities>\n      <Limits SoftLimit="255" HardLimit="255"/>\n   </MaltegoTransformRequestMessage>\n</MaltegoMessage>\n'
        node: <DOM Element: MaltegoMessage at 0x7f156a955290>
        attrs: []
        Current node has the following defined fields:
                field: <safedexml.fields.Choice object at 0x7f156a9580d0> (tagname: None) (attrname: None) (field name: message) (model class: <class 'canari.maltego.message.MaltegoMessage'>)
                         Has unused_attrs: []
        parsing order_sensitive = True
        parse(....) ------------------------------------------------------------------------------------
                Constructing class: MaltegoTransformRequestMessage
                Generating node from XML: <DOM Element: MaltegoTransformRequestMessage at 0x7f156a955368>
                node: <DOM Element: MaltegoTransformRequestMessage at 0x7f156a955368>
                attrs: []
                Current node has the following defined fields:
                        field: <safedexml.fields.List object at 0x7f156a94ea50> (tagname: Entities) (attrname: None) (field name: entities) (model class: <class 'canari.maltego.message.MaltegoTransformRequestMessage'>)
                                 Has unused_attrs: []
                        field: <safedexml.fields.Dict object at 0x7f156a94eed0> (tagname: TransformFields) (attrname: None) (field name: parameters) (model class: <class 'canari.maltego.message.MaltegoTransformRequestMessage'>)
                                 Has unused_attrs: []
                        field: <safedexml.fields.Model object at 0x7f156a94ef50> (tagname: None) (attrname: None) (field name: limits) (model class: <class 'canari.maltego.message.MaltegoTransformRequestMessage'>)
                                 Has unused_attrs: []
                parsing order_sensitive = True
                parse(....) ------------------------------------------------------------------------------------
                        Constructing class: Limits
                        Generating node from XML: <DOM Element: Limits at 0x7f156a955e18>
                        node: <DOM Element: Limits at 0x7f156a955e18>
                        attrs: [<xml.dom.minidom.Attr instance at 0x7f156a955ea8>, <xml.dom.minidom.Attr instance at 0x7f156a955f80>]
                        Current node has the following defined fields:
                                field: <safedexml.fields.Integer object at 0x7f156a940d90> (tagname: None) (attrname: SoftLimit) (field name: soft) (model class: <class 'canari.maltego.message.Limits'>)
                                         Has unused_attrs: [<xml.dom.minidom.Attr instance at 0x7f156a955f80>]
                                         adding field to list of found
                                field: <safedexml.fields.Integer object at 0x7f156a940dd0> (tagname: None) (attrname: HardLimit) (field name: hard) (model class: <class 'canari.maltego.message.Limits'>)
                                         Has unused_attrs: []
                                         adding field to list of found
                        parsing order_sensitive = True
                        parse() check that required fields have been found
                                 looping field: <safedexml.fields.Integer object at 0x7f156a940d90>
                                 looping field: <safedexml.fields.Integer object at 0x7f156a940dd0>
                        Created: Limits
                parse(....) ----------------------------------------- END --------------------------------------
                parse() check that required fields have been found
                         looping field: <safedexml.fields.List object at 0x7f156a94ea50>
ERROR: An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line string', (1, 0))

---------------------------------------------------------------------------
Removal of dead code and elimination of a bug.

Dead code:

  self._canari_fields = dict([(f.name, f.value) for f in  self.entity.fields.values()])

We can remove this line of code from the constructor of
MaltegoTransformRequestMessage, as it is basically a no op:

 1. Trying to read from self.entity in __init__ will ALWAYS end
    up returning a 'none-entity' (return Entity('')), as the read
    from self.entities will fail due to the fact that it has not
    been populated with anything yet.

    a. This read from self.entities has the side effect of
       reading from the safedexml List field type, which
       initialises it to the empty list.  When we later on tries
       to actually parse anything into the field, it will fail as
       the call:

          val = super(List, self).__get__(obj)

       inside safedexml.fields.List.parse_child_node will return
       '[]' instead of None.  Because of this, the if statement
       will skip the 'parsing logic' and eventually end up
       returning PARSE_SKIP, instead of PARSE_CHILDREN.

    b. This has the effect that the <entities>...</entities> tag
       is not parsed, and thus the entities List field of the
       request message is newer populated.

 2. '_canari_fields' is not references anywhere else in the canari code.
    So even if the call did return anything, it would newer be used.

 3. To the best of my knowledge the code doesn't produce any side
    effects that are needed or even desired.

With the dead code, the following valid XML from Maltego, will
not populate the entities field.  This is an issue, as the
request message is sent to the transform, from where it is not
possible to access any of the information about the entity (most
importantly the value).
Worse is, if the previous bugfix has been implemented, where the
entities is no longer optional.  Then it will fail with an
ParseError due to the required field not being found.

--------------------------------------------------------

In [1]: import canari.maltego.message as msg

In [2]: mmsg = msg.MaltegoMessage.parse('''
<MaltegoMessage>
   <MaltegoTransformRequestMessage>
      <Entities>
         <Entity Type="Domain">
            <AdditionalFields>
               <Field Name="fqdn" DisplayName="Domain
               Name">paterva.com</Field>
            </AdditionalFields>
            <Weight>0</Weight>
            <Value>paterva.com</Value>
         </Entity>
      </Entities>
      <Limits SoftLimit="255" HardLimit="255"/>
   </MaltegoTransformRequestMessage>
</MaltegoMessage>
''')

--------------------------------------------------------

Now if trying to access the entity property of the request
message, we will get a non-entity back as seen by the empty
value and the empty list of entities.

--------------------------------------------------------

In [3]: mmsg.message.entity.value
Out[3]: ''

In [4]: mmsg.message.entities
Out[4]: []

--------------------------------------------------------

This can be furhter documented by adding debug statements inside
the 'entity' property.  However with the dead code eliminated,
the correct information is returned as seen below.

--------------------------------------------------------

In [3]: mmsg.message.entity.value
Out[3]: u'paterva.com'

In [4]: mmsg.message.entities
Out[4]: [<canari.maltego.message._Entity at 0x7f1ef1b1ff50>]

--------------------------------------------------------

All in all, this seem to be a bug in safedexml.  Currently I have
only noticed it on the List field type, however it may or may not
be present on other field types as well.  This isue is solved by
not reading from the entities field before the user has a chance
to call the parse function or if explicitly adding instances into
it (by appendelement) for use when rendering to xml.
Merge branch 'fix/entities-are-not-optional-in-MaltegoTransformRequestMessage' into feature/maltego-message