Add a database table for raw understand metrics

Table understand_raw stores the (molten) data frame obtained
from the complexity analysis done by understand. Simpler univariate
time series can then be derived from this representation.

Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Add an R interface to understand

Understand is a complexity analysis tool that allows us to
obtain various pieces of statistical information of interest
to Codeface. Currently, acquiring these data is done via another
external project (ChangeAnalysis respectively iFest) , but it has
turned out that the approach causes substantial performance issues.
Provide a light-weight R wrapper to the functionality offered
by understand that uses sampling techniques to obtain estimates
of the time series of the various metrics supported by understand.

Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Handle single-core multiprocessing case correctly

Make sure to also set conf$jobs when only a single core
is used. This fixes some invalid messages like "Using  jobs"
without an explicit core count.

Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Wire sloccount analysis into complexity analysis pass

Since the complexity analysis pass provides all the infrastructure
that is required for the sloccount analysis (iteration over intermediate
states of the repository), integrate the two passes.

Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Change visual layout of database model

Group related components together and add layers. No functional
changes, therefore no SQL script update required.

Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Fix foreign keys for per_cluster_statistics

The table does not contain foreign keys for projectID
and releaseRangeId, although these are required to ensure
consistency, and to implement the chain deletion mechanism.
Add them.

Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Remove quantarch references from database model

"quantarch" used to be an old, internal name for the project.
This is now fully replaced with codeface, so update the database
schema accordingly. Also adapt the default configuration file
to the new naming convention.

Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Properly handle empty sloccount return values

Handle the case when sloccount does not find any source
files known to the program by making the regexp analysis
more robust.

Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Fix keys/indices in understand_raw

There are obviously multiple data points in a plot,
so using plotId as primary key does obviously not make
sense because the entry is highly non-unique. For the same
reasons, it is counter-productive to require unique timestamps.
Use non-unique indices for both columns.

Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Rename removeOutliers() to remove.outliers()

This make the function name comply to our standard naming
scheme.

Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Equip plots with labels

The dimension of plotted quantities can be ambiguous, so
add two fields to table plots to describe the meaning.
Update writers to table timeseries.

Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>

fixup
Post-process sloccount time series fragments

Add a new analysis pass to analyse_ts that takes
the raw sloccount information computed by complexity.r,
and create time series that can be immediately plotted.

Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Pre-process raw understand statistics before db serialisation

A typical understand run on an average to large project (for instance,
u-boot or qemu) produces nearly a million data points that have
previously been stored in table understand_raw. This is way too much
for a substantial history; reduce the data set to all pieces information
required to produce a boxplot.

Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Improve complexity measurement logging

Explicitly say which revision is checked out, and not
just into which directory the checkout was performed.
Since temporaries are cleaned up after analysis, this
information is not helpful. Using the hash, we can
easily reconstruct failed states.

Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Make index handling for clusters more explicit

We rely on the fact that vertices of clusters represented
in igraph objects have consecutive indices starting from 1,
but this is currently only true because of a side effeft (column
names get lost when the adjacency matrix is transposed in
performAnalysis). Since consecutive labels are a) assumed by
the rest of the code and b) required for some graph operations,
ensure that the property is actively enforced.

Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Fix some formal issues in person.r

Correct faulty indentation, and remove a typo in a comment.

Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Do not overwrite commit date in complexity analysis

The commit date is taken from the list of commits, but later
on incidentally overwritten with the date from the understand
analysis. However, understand can fail in some inputs, which
leads to good dates being transformed into empty dates, causing
SQL problems in turn later. Fix this by not setting a potentially
bogus date at all.

Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Fix handling invalid understand results

If understand does not produce results for one reason
or another, parse.understand() returns NULL. While the
subset selection on a NULL object succeeds, the test
for the number of columns does not -- it leads to a crash
instead of skipping this result set. Properly check
for invalid results by using a check for NULL, thus.

Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Ignore understand raw data in complexity analysis widget

Currently, the complexity analysis widget assumes that all
plots with a name like "understand*" represent time series,
which is not true anymore with the addition of raw data.
Correct this assumption to fix all widgets that rely on the
old behaviour.

Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>
Improve installation instructions

Update the lists of required packages, and provide some more
background information on why specific dependencies are
needed.

Signed-off-by: Wolfgang Mauerer <wolfgang.mauerer@siemens.com>