Cover empty intervals in complexity analysis

Very small release ranges (bugfix or formal releases with just
a tag) may contain no clusters. Handle this case properly.

Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Skip sloccount analysis when the analysis pass is not configured

When no sloccount data are collected, there's nothing that
can be analysed.

Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Improve complexity analysis console output

Give less, but more useful information. Restrict large
volume outputs to log level devinfo.

Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Fix ML time series data frame creation

Plot labels are not stored with the data as such, but
in the plot description. Fix construction of the respective
data frame.

Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Vagrantfile: initial version with basic setup

Signed-off-by: Roger Meier <roger@bufferoverflow.ch>
Vagrantfile: setup db, recent node.js, forward 8081

Signed-off-by: Roger Meier <roger@bufferoverflow.ch>
packages.R: add shiny-gridster

Signed-off-by: Roger Meier <roger@bufferoverflow.ch>
add mailing list analysis dependencies for R

Signed-off-by: Roger Meier <roger@bufferoverflow.ch>
add missing dependencies
- Debian Package: graphviz
- R Packages: rjson, lsa

Signed-off-by: Roger Meier <roger@bufferoverflow.ch>
Ensure that the node.js binary is available as nodejs

Debian-like official distributions like Ubuntu 14.04 install the
node.js binary under a different name than provided by upstream
node.js. Document how to set an alias with update-alternatives.

Signed-off-by: Gottfried Hoffmann <gottfried@blenderdiplom.com>
Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Fix: python-mysql library requires iterable types

- python-mysql requires the queries to be formated like so
  "SELECT id FROM project WHERE name=%s "
    "AND analysisMethod=%s", (name, analysisMethod)

- when the query only has one variable a tuple must still be
  otherwise execution failes
  e.g., ("SELECT id FROM project WHERE name=%s;", (name,))
    not ("SELECT id FROM project WHERE name=%s;", name)

Signed-off-by: Mitchell Joblin <mitchell.joblin.ext@siemens.com>
Reviewed-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Fix mismatch between dev ids and adjacency matrix columns

- When querying the database for developer ids, the results
  does not necessarily match the order of ids in the
  adjacency matrix, yet we make this assumption in later code

- We now encode the adjacency matrix header with developer ids
  instead of names, then we compare the ids we get from the
  database with the ids we parse from the adjacency matrix
  storage file

- If a mismatch exists we remap the order correctly

Signed-off-by: Mitchell Joblin <mitchell.joblin.ext@siemens.com>
Reviewed-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Add integration test for network construction correctness

- None of the existing tests check that the network is being
  constructed correctly from the commit data

- The test will query the database after the analysis pass for
  the network and the person ids, it will then verify that the
  network is correctly linking developers and with the correct
  weight by comparing to known correct network for a given
  set of commits and construction method

- The test will help us to identify any problems with the
  developer ids and the reference to the network, in addition
  to verifying that the entire network construction process
  is functioning properly

- The test is currently only implemented for the proximity
  based analysis, for full coverage we need to calculate
  a ground truth network for the other analysis methods

Signed-off-by: Mitchell Joblin <mitchell.joblin.ext@siemens.com>
Reviewed-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Change mechanism to parse sign-off tags and add apache style tags

- We now use regular expressions rather than using the string
  function "startswith"

- Additional tags are added which address how apache projects
  annotate the commit message

- Another benefit of this mechanism of matching is that we match
  without case sensitivity, this is especially imporant for apache style
  tags where I noticed variants of Patch, patch and PATCH tags
  which makes the case matching annoying and using tolower(..)
  would reduce performance

- We now also get rid of the massive if conditional and consolidate
  the patterns

- Change parser to exactly match our tag patterns instead of generic
  sign-off pattern

- apache style tags are less strictly formed which demands
  parsing them differently than how we handles kernel tags
  for example there can be an arbitray number of white space
  characters between tag and name and we must check entire
  commit description for sign off tags not just
  the last line because often times tags are not found only
  on the last line, again using regular expressions should
  reduce the performace decrease associated with  parsing
  the entire commit description

Signed-off-by: Mitchell Joblin <mitchell.joblin.ext@siemens.com>
Bug Fix: statement incorrectly in loop scope

- seems like the commit message does not need to be inside
  the loop and it means that we are double counting tags
  and wasting cycles on parsing the same message twice

Signed-off-by: Mitchell Joblin <mitchell.joblin.ext@siemens.com>
Add Apache Thrift configuration file

Signed-off-by: Mitchell Joblin <mitchell.joblin.ext@siemens.com>
Fix: Id case where no email address is found for a developer

- this case happens when a developer does not add there
  email in a commit

- some projects are very strict about always including a developer
  email but in the case of apache style tags the email is most
  often not included, only the developer name is

- developers without an email were previously assigned a generic replacement,
  the problem with this approach is that the email is used for
  merging aliases and thus everyone with the same email gets assigned
  the same person Id

- we now introduce a random string email replacement so that the
  alias merging will resolve unique Ids for people without an
  email address

- an alternative solution would be to change the order in which
  the id service performes alias merging and keep the generic
  replacement email, I think that the generic email is a bad
  idea in general since the nature of an email address is
  unique to the individual or organization that it represents
  and we destroy that notion of uniqueness when we assign a generic
  email, I think we still need to know when an email was artificially
  generated and thats why we need a generic plus a unique component
  for the email as provided by this patch

Signed-off-by: Mitchell Joblin <mitchell.joblin.ext@siemens.com>
Set R environment locale to english

- if the locale is not english the parser will silently incorrectly
  parse emails

Signed-off-by: Mitchell Joblin <mitchell.joblin.ext@siemens.com>
Fix: Three digit group indexing is too small for large projects

- Now index groups with 4 digits

- previously if the number exceeds three digits an error caused
  execution to fail

- In the case of large projects like the linux kernel we can
  exceed the previous 3 digit limit

Signed-off-by: Mitchell Joblin <mitchell.joblin.ext@siemens.com>
Check corpus for precondition violations

- later on in the analysis there are several conditions which
  must be satified otherwise a failure can result

- we introduce a new function where all preconditions can be checked
  for and problematic documents are removed from the corpus

- the particular preconditions added here prevent failures
  from occuring in the threads(..) function in the tm.plugin.mail
  library which fails when multiple reference Ids are found in
  an email

- In the case where multiple reference Ids are found we keep the
  first reference Id and discard the remaining references by manipulating
  the corpus

Signed-off-by: Mitchell Joblin <mitchell.joblin.ext@siemens.com>