Package: sirmordred
SirMordredSirMordred is the tool used to coordinate the execution of the GrimoireLab platform, via two main configuration files, thesetup.cfgandprojects.json, which are summarized in their corresponding sections.ContentsSetup.cfgProjects.jsonSupported data sourcesMicro MordredGetting startedTroubleshootingHow toSetup.cfg↑The setup file holds the configuration to arrange all processes underlying GrimoireLab. It is composed of sections that allow defining the general settings such as which phases to activate (e.g., collection, enrichment) and where to store the logs, as well as the location and credentials for SortingHat and the ElasticSearch instances where the raw and enriched data is stored. Furthermore, it also includes backend sections to set up the parameters used by Perceval to access the software development tools (e.g., GitHub tokens, Gerrit username) and fetch their data.Dashboards can be automatically uploaded via thesetup.cfgif the phasepanelsis enabled. TheData StatusandOverviewdashboards will contain
widgets that summarize the information of the data sources declared in thesetup.cfg. Note that the widgets are not updated when adding
new data sources, thus you need to manually delete the dashboardsData StatusandOverview(underStack Management > Saved Objectsin Kibiter), and restart mordred again (making sure that the optionpanelsis enabled).[es_collection]url(str:http://172.17.0.1:9200): Elasticsearch URL (Required)[es_enrichment]autorefresh(bool: True): Execute the autorefresh of identitiesautorefresh_interval(int: 2): Time interval (days) to autorefresh identitiesurl(str:http://172.17.0.1:9200): Elasticsearch URL (Required)[general]bulk_size(int: 1000): Number of items to write in Elasticsearch using bulk operationsdebug(bool: True): Debug mode (logging mainly) (Required)logs_dir(str: logs): Directory with the logs of sirmordred (Required)min_update_delay(int: 60): Short delay between tasks (collect, enrich ...)scroll_size(int: 100): Number of items to read from Elasticsearch when scrollingshort_name(str: Short name): Short name of the project (Required)update(bool: False): Execute the tasks in loop (Required)aliases_file(str: ./aliases.json): JSON file to define aliases for raw and enriched indexesmenu_file(str: ./menu.yaml): YAML file to define the menus to be shown in Kibiterglobal_data_sources(list: bugzilla, bugzillarest, confluence, discourse, gerrit, jenkins, jira): List of data sources collected globally, they are declared in the section 'unknown' of the projects.jsonretention_time(int: None): the maximum number of minutes wrt the current date to retain the dataupdate_hour(int: None): The hour of the day the tasks will run ignoringmin_update_delay(collect, enrich ...)[panels]community(bool: True): Include community section in dashboardkibiter_default_index(str: git): Default index pattern for Kibiterkibiter_time_from(str: now-90d): Default time interval for Kibiterkibiter_url(str): Kibiter URL (Required)kibiter_version(str: None): Kibiter versionkafka(bool: False): Include KIP section in dashboardgithub-comments(bool: False): Enable GitHub comments menu. Note that if enabled, the github2:issue and github2:pull sections in the setup.cfg and projects.json should be declaredgithub-events(bool: False): Enable GitHub events menu. Note that if enabled, the github:event section in the setup.cfg and projects.json should be declaredgithub-repos(bool: False): Enable GitHub repo stats menu. Note that if enabled, the github:repo section in the setup.cfg and projects.json should be declaredgitlab-issues(bool: False): Enable GitLab issues menu. Note that if enabled, the gitlab:issue section in the setup.cfg and projects.json should be declaredgitlab-merges(bool: False): Enable GitLab merge requests menu. Note that if enabled, the gitlab:merge sections in the setup.cfg and projects.json should be declaredmattermost(bool: False): Enable Mattermost menucode-license(bool: False): Enable code license menu. Note that if enabled, colic sections in the setup.cfg and projects.json should be declaredcode-complexity(bool: False): Enable code complexity menu. Note that if enabled, cocom sections in the setup.cfg and projects.json should be declaredstrict(bool: True): Enable strict panels loadingcontact(str: None): Support repository URL[phases]collection(bool: True): Activate collection of items (Required)enrichment(bool: True): Activate enrichment of items (Required)identities(bool: True): Do the identities tasks (Required)panels(bool: True): Load panels, create alias and other tasks related (Required)[projects]projects_file(str: projects.json): Projects file path with repositories to be collected grouped by projectsprojects_url(str: None): Projects file URL, the projects_file is required to store the file locally[sortinghat]affiliate(bool: True): Affiliate identities to organizations (Required)autogender(bool: False): Add gender to the profiles (executes autogender)autoprofile(list: ['customer', 'git', 'github']): Order in which to get the identities information for filling the profile (Required)database(str: sortinghat_db): Name of the Sortinghat database (Required)host(str: 127.0.0.1):  Host with the Sortinghat database (Required)port(int: None):  GraphQL server portpath(str: None) GraphQL pathssl(bool: False) GraphQL server use SSL/TSL connectionmatching(list: ['email']): Algorithm for matching identities in Sortinghat (Required)password(str: ): Password to access the Sortinghat database (Required)reset_on_load(bool: False): Unmerge and remove affiliations for all identities on loadsleep_for(int: 3600): Delay between task identities executions (Required)strict_mapping(bool: True): rigorous check of values in identities matching (i.e, well formed email addresses, non-overlapping enrollment periods)unaffiliated_group(str: Unknown): Name of the organization for unaffiliated identities (Required)user(str: root): User to access the Sortinghat database (Required)[backend-name:tag] (tag is optional)collect(bool: True): enable/disable collection phaseraw_index(str: None): Index name in which to store the raw items (Required)enriched_index(str: None): Index name in which to store the enriched items (Required)studies(list: []): List of studies to be executedanonymize(bool: False): enable/disable anonymization of personal user informationbackend-param-1: ..backend-param-2: ..backend-param-n: ..The template of a backend section is shown above.
Further information about Perceval backends parameters are available at:Params details:https://perceval.readthedocs.io/en/latest/perceval/perceval-backends.htmlExamples:https://github.com/chaoss/grimoirelab-sirmordred/blob/master/tests/test_studies.cfgNote that some backend sections allow to specify specific enrichment options, which are listed below.[jenkins]node_regex: regular expression for extracting node name frombuiltOnfield. This
regular expressionmust contain at least one group. First group will be used to extract
node name. More groups are allowed but not used to extract anything else.[studies-name:tag] (tag is optional)study-param-1: ..study-param-2: ..study-param-n: ..A template of a study section is shown above. A complete list of studies parameters is available at:https://github.com/chaoss/grimoirelab-sirmordred/blob/master/tests/test_studies.cfgProjects.json↑The projects.json aims at describing the repositories grouped by a project that will be shown on the dashboards.The project file enables the users to list the instances of the software development tools to analyse, such
as local and remote Git repositories, the URLs of GitHub and GitLab issue trackers, and the name of Slack channels.
Furthermore, it also allows the users to organize these instances into nested groups, which structure is reflected in
the visualization artifacts (i.e., documents and dashboards). Groups can be useful to represent projects within a single
company, sub-projects within a large project such as Linux and Eclipse, or the organizations within a collaborative project.First level: project namesSecond level: data sources and metadataThird level: data source URLsThere are some filters, labels, and a special section:Filter--filter-no-collection=true: This filter is used to show old enriched data within the dashboards from
repositories that don't exist anymore in upstream.Filter--filter-rawand the sectionunknown: The data sources will only collected at the sectionunknownbut this allow to add the same source in different sections to enrich using the filter--filter-raw.Label--labels=[example]: The data source will have the label ofexamplewhich can be used to create visualisations for specific sets of dataSectionunknown: If the data source is only under this section it will be enriched as projectmain.{
    "Chaoss": {
        "gerrit": [
            "gerrit.chaoss.org --filter-raw=data.projects:CHAOSS"
        ]
        "git": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https://<username>:<api-token>@github.com/chaoss/grimoirelab-sirmordred"
        ],
        "github": [
            "https:/github.com/chaoss/grimoirelab-perceval --filter-no-collection=true",
            "https:/github.com/chaoss/grimoirelab-sirmordred  --labels=[example]"
        ]
    },
    "GrimoireLab": {
        "gerrit": [
            "gerrit.chaoss.org --filter-raw=data.projects:GrimoireLab"
        ],
        "meta": {
            "title": "GrimoireLab",
            "type": "Dev",
            "program" : "Bitergia",
            "state": "Operating"
        },
    }
    "unknown": {
        "gerrit": [
            "gerrit.chaoss.org"
        ],
        "confluence": [
            "https://wiki.chaoss.org"
        ]
}In the projects.json above:The data included in the repogerrit.chaoss.orgwill be collected entirely since the
repo is listed in theunknownsection. However only the projectGrimoireLabwill be enriched as declared in theGrimoireLabsection.In the sectionChaossin the data sourcegithubthe repositorygrimoirelab-percevalis not collected for
raw index but it will enriched in the enriched index.In the sectionGrimoireLabthe metadata will showed in the enriched index as extra fields.In the sectionunknownthe data sourceconfluencewill be enriched as the projectmain.Supported data sources↑These are the data sources GrimoireLab supports:askbot,bugzilla,bugzillarest,cocom,colic,confluence,crates,discourse,dockerhub,dockerdeps,dockersmells,functest,gerrit,git,github,github2,gitlab,gitter,google_hits,groupsio,hyperkitty,jenkins,jira,kitsune,mattermost,mbox,mediawiki,meetup,mozillaclub,nntp,pagure,phabricator,pipermail,puppetforge,redmine,remo,rocketchat,rss,slack,stackexchange,supybot,telegram,twitteraskbot↑Questions and answers from Askbot siteprojects.json{
    "Chaoss": {
        "askbot": [
            "https://askbot.org/"
        ]
    }
}setup.cfg[askbot]
raw_index = askbot_raw
enriched_index = askbot_enrichedbugzilla↑Bugs from Bugzillaprojects.json{
    "Chaoss": {
        "bugzilla": [
            "https://bugs.eclipse.org/bugs/"
        ]
    }
}setup.cfg[bugzilla]
raw_index = bugzilla_raw
enriched_index = bugzilla_enriched
backend-user = yyyy (optional)
backend-password = xxxx (optional)
no-archive = true (suggested)bugzillarest↑Bugs from Bugzilla server (>=5.0) using its REST APIprojects.json{
    "Chaoss": {
        "bugzillarest": [
            "https://bugzilla.mozilla.org"
        ]
    }
}setup.cfg[bugzillarest]
raw_index = bugzillarest_raw
enriched_index = bugzillarest_enriched
backend-user = yyyy (optional)
backend-password = xxxx (optional)
no-archive = true (suggested)cocom↑Code complexity integration.
Some graal dependencies likeclocmight be required,https://github.com/chaoss/grimoirelab-graal#how-to-installcreate-the-executablesprojects.json{
    "Chaoss":{
        "cocom": [
            "https://github.com/chaoss/grimoirelab-toolkit"
        ]
    }
}setup.cfg[cocom]
raw_index = cocom_chaoss
enriched_index = cocom_chaoss_enrich
category = code_complexity_lizard_file
studies = [enrich_cocom_analysis]
branches = master
worktree-path = /tmp/cocom/colic↑Code license backend.projects.json{
    "Chaoss":{
        "colic": [
            "https://github.com/chaoss/grimoirelab-toolkit"
        ]
    }
}setup.cfg[colic]
raw_index = colic_chaoss
enriched_index = colic_chaoss_enrich
category = code_license_nomos
studies = [enrich_colic_analysis]
exec-path = /usr/share/fossology/nomos/agent/nomossa
branches = master
worktree-path = /tmp/colicconfluence↑contents from Confluenceprojects.json{
    "Chaoss": {
        "confluence": [
            "https://wiki.open-o.org/"
        ]
    }
}setup.cfg[confluence]
raw_index = confluence_raw
enriched_index = confluence_enriched
no-archive = true (suggested)crates↑packages from Crates.ioprojects.json{
    "Chaoss": {
        "crates": [
            ""
        ]
    }
}setup.cfg[crates]
raw_index = crates_raw
enriched_index = crates_enricheddiscourse↑Topics from Discourseprojects.json{
    "Chaoss": {
        "discourse": [
            "https://foro.mozilla-hispano.org/"
        ]
    }
}setup.cfg[discourse]
raw_index = discourse_raw
enriched_index = discourse_enriched
no-archive = true (suggested)dockerhub↑Repositories info from DockerHubprojects.json{
    "Chaoss": {
        "dockerhub": [
            "bitergia kibiter"
        ]
    }
}setup.cfg[dockerhub]
raw_index = dockerhub_raw
enriched_index = dockerhub_enriched
no-archive = true (suggested)dockerdeps↑Dependencies extracted from Dockerfiles. Requireshttps://github.com/crossminer/crossJadolintprojects.json{
    "Chaoss": {
        "dockerdeps": [
            "https://github.com/chaoss/grimoirelab"
        ]
    }
}setup.cfg[dockerdeps]
raw_index = dockerdeps_raw
enriched_index = dockerdeps_enrich
category = code_dependencies_jadolint
exec-path = <jadolint-local-path>/jadolint.jar
in-paths = [Dockerfile, Dockerfile-full, Dockerfile-secured, Dockerfile-factory, Dockerfile-installed]dockersmells↑Smells extracted from Dockerfiles. Requireshttps://github.com/crossminer/crossJadolintprojects.json{
    "Chaoss": {
        "dockersmells": [
            "https://github.com/chaoss/grimoirelab"
        ]
    }
}setup.cfg[dockersmells]
raw_index = dockersmells_raw
enriched_index = dockersmells_enrich
category = code_quality_jadolint
exec-path = <jadolint-local-path>/jadolint.jar
in-paths = [Dockerfile, Dockerfile-full, Dockerfile-secured, Dockerfile-factory, Dockerfile-installed]functest↑Tests from functestprojects.json{
    "Chaoss": {
        "functest": [
            "http://testresults.opnfv.org/test/"
        ]
    }
}setup.cfg[functest]
raw_index = functest_raw
enriched_index = functest_enriched
no-archive = true (suggested)gerrit↑Reviews from GerritYou have to add your public key in the gerrit server.projects.json{
    "Chaoss": {
        "gerrit": [
            "review.opendev.org"
        ]
    }
}setup.cfg[gerrit]
raw_index = gerrit_raw
enriched_index = gerrit_enriched
user = xxxx
no-archive = true (suggested)
blacklist-ids = [] (optional)
max-reviews = 500 (optional)
studies = [enrich_demography:gerrit, enrich_onion:gerrit, enrich_demography_contribution:gerrit] (optional)

[enrich_demography:gerrit] (optional)

[enrich_onion:gerrit] (optional)
in_index = gerrit_enriched
out_index = gerrit-onion_enriched

[enrich_demography_contribution:gerrit] (optional)
date_field = grimoire_creation_date
author_field = author_uuidgit↑Commits from GitNote:If you want to analyze private git repositories, make sure you pass the credentials directly in the URL.projects.json{
    "Chaoss": {
        "git": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https://<username>:<api-token>@github.com/chaoss/grimoirelab-sirmordred"
        ]
    }
}setup.cfg[git]
raw_index = git_raw
enriched_index = git_enriched
latest-items = true (suggested)
studies = [enrich_demography:git, enrich_git_branches:git, enrich_areas_of_code:git, enrich_onion:git, enrich_extra_data:git] (optional)

[enrich_demography:git] (optional)

[enrich_git_branches:git] (optional)
run_month_days = [1, 23] (optional)

[enrich_areas_of_code:git] (optional)
in_index = git_raw
out_index = git-aoc_enriched

[enrich_onion:git] (optional)
in_index = git_enriched
out_index = git-onion_enriched

[enrich_extra_data:git]
json_url = https://gist.githubusercontent.com/zhquan/bb48654bed8a835ab2ba9a149230b11a/raw/5eef38de508e0a99fa9772db8aef114042e82e47/bitergia-example.txt

[enrich_forecast_activity]
out_index = git_study_forecastgithub↑Issues and PRs from GitHubissueprojects.json{
    "Chaoss": {
        "github:issue": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https:/github.com/chaoss/grimoirelab-sirmordred"
        ]
    }
}setup.cfg[github:issue]
raw_index = github_raw
enriched_index = github_enriched
api-token = xxxx
category = issue
sleep-for-rate = true
no-archive = true (suggested)
studies = [enrich_onion:github,
           enrich_geolocation:user,
           enrich_geolocation:assignee,
           enrich_extra_data:github,
           enrich_backlog_analysis,
           enrich_demography:github] (optional)

[enrich_onion:github] (optional)
in_index_iss = github_issues_onion-src
in_index_prs = github_prs_onion-src
out_index_iss = github-issues-onion_enriched
out_index_prs = github-prs-onion_enriched

[enrich_geolocation:user] (optional)
location_field = user_location
geolocation_field = user_geolocation

[enrich_geolocation:assignee] (optional)
location_field = assignee_location
geolocation_field = assignee_geolocation

[enrich_extra_data:github]
json_url = https://gist.githubusercontent.com/zhquan/bb48654bed8a835ab2ba9a149230b11a/raw/5eef38de508e0a99fa9772db8aef114042e82e47/bitergia-example.txt

[enrich_backlog_analysis]
out_index = github_enrich_backlog
interval_days = 7
reduced_labels = [bug,enhancement]
map_label = [others, bugs, enhancements]

[enrich_demography:github]pull requestprojects.json{
    "Chaoss": {
        "github:pull": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https:/github.com/chaoss/grimoirelab-sirmordred"
        ]
    }
}setup.cfg[github:pull]
raw_index = github-pull_raw
enriched_index = github-pull_enriched
api-token = xxxx
category = pull_request
sleep-for-rate = true
no-archive = true (suggested)
studies = [enrich_geolocation:user,
           enrich_geolocation:assignee,
           enrich_extra_data:github,
           enrich_demography:github] (optional)

[enrich_geolocation:user]
location_field = user_location
geolocation_field = user_geolocation

[enrich_geolocation:assignee]
location_field = assignee_location
geolocation_field = assignee_geolocation

[enrich_extra_data:github]
json_url = https://gist.githubusercontent.com/zhquan/bb48654bed8a835ab2ba9a149230b11a/raw/5eef38de508e0a99fa9772db8aef114042e82e47/bitergia-example.txt

[enrich_demography:github]repoThe number of forks, stars, and subscribers in the repository.projects.json{
    "Chaoss": {
        "github:repo": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https:/github.com/chaoss/grimoirelab-sirmordred"
        ]
    }
}setup.cfg[github:repo]
raw_index = github-repo_raw
enriched_index = github-repo_enriched
api-token = xxxx
category = repository
sleep-for-rate = true
no-archive = true (suggested)
studies = [enrich_extra_data:github, enrich_demography:github]

[enrich_extra_data:github]
json_url = https://gist.githubusercontent.com/zhquan/bb48654bed8a835ab2ba9a149230b11a/raw/5eef38de508e0a99fa9772db8aef114042e82e47/bitergia-example.txt

[enrich_demography:github]githubql↑Events from GitHubThe corresponding dashboards can be automatically uploaded by settinggithub-eventstotruein thepanelssection within thesetup.cfgprojects.json{
    "Chaoss": {
        "githubql": [
            "https://github.com/chaoss/grimoirelab-toolkit"
        ]
    }
}setup.cfg[panels]
github-events = true

[githubql]
raw_index = github_event_raw
enriched_index = github_event_enriched
api-token = xxxxx
sleep-for-rate = true
sleep-time = "300" (optional)
no-archive = true (suggested)
studies = [enrich_duration_analysis:kanban, enrich_reference_analysis] (optional)

[enrich_duration_analysis:kanban]
start_event_type = MovedColumnsInProjectEvent
fltr_attr = board_name
target_attr = board_column
fltr_event_types = [MovedColumnsInProjectEvent, AddedToProjectEvent]

[enrich_duration_analysis:label]
start_event_type = UnlabeledEvent
target_attr = label
fltr_attr = label
fltr_event_types = [LabeledEvent]

[enrich_reference_analysis] (optional)github2↑Comments from GitHubThe corresponding dashboards can be automatically uploaded by settinggithub-commentstotruein thepanelssection within thesetup.cfgissueprojects.json{
    "Chaoss": {
        "github2:issue": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https:/github.com/chaoss/grimoirelab-sirmordred"
        ]
    }
}setup.cfg[github2:issue]
api-token = xxxx
raw_index = github2-issues_raw
enriched_index = github2-issues_enriched
sleep-for-rate = true
category = issue
no-archive = true (suggested)
studies = [enrich_geolocation:user, enrich_geolocation:assignee, enrich_extra_data:github2, enrich_feelings] (optional)

[enrich_geolocation:user] (optional)
location_field = user_location
geolocation_field = user_geolocation

[enrich_geolocation:assignee] (optional)
location_field = assignee_location
geolocation_field = assignee_geolocation

[enrich_extra_data:github2]
json_url = https://gist.githubusercontent.com/zhquan/bb48654bed8a835ab2ba9a149230b11a/raw/5eef38de508e0a99fa9772db8aef114042e82e47/bitergia-example.txt

[enrich_feelings]
attributes = [title, body]
nlp_rest_url = http://localhost:2901pull requestprojects.json{
    "Chaoss": {
        "github2:pull": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https:/github.com/chaoss/grimoirelab-sirmordred"
        ]
    }
}setup.cfg[github2:pull]
api-token = xxxx
raw_index = github2-pull_raw
enriched_index = github2-pull_enriched
sleep-for-rate = true
category = pull_request
no-archive = true (suggested)
studies = [enrich_geolocation:user, enrich_geolocation:assignee, enrich_extra_data:git, enrich_feelings] (optional)

[enrich_geolocation:user] (optional)
location_field = user_location
geolocation_field = user_geolocation

[enrich_geolocation:assignee] (optional)
location_field = assignee_location
geolocation_field = assignee_geolocation

[enrich_extra_data:github2]
json_url = https://gist.githubusercontent.com/zhquan/bb48654bed8a835ab2ba9a149230b11a/raw/5eef38de508e0a99fa9772db8aef114042e82e47/bitergia-example.txt

[enrich_feelings]
attributes = [title, body]
nlp_rest_url = http://localhost:2901gitlab↑Issues and MRs from GitLabGitLab issues and merge requests need to be configured in two different sections.
The corresponding dashboards can be automatically uploaded by settinggitlab-issueandgitlab-mergetotruein thepanelssection within thesetup.cfgIf a given GitLab repository is under more than 1 level, all the slashes/starting from the second level have to be
replaced by%2F. For instance, for a repository with a structure similar to this onehttps://gitlab.com/Molly/lab/first.issueprojects.json{
    "Chaoss": {
        "gitlab:issue": [
            "https://gitlab.com/Molly/first",
            "https://gitlab.com/Molly/lab%2Fsecond"
        ]
    }
}setup.cfg[panels]
gitlab-issues = true

[gitlab:issue]
category = issue
raw_index = gitlab-issues_raw
enriched_index = gitlab-issues_enriched
api-token = xxxx
sleep-for-rate = true
no-archive = true (suggested)
studies = [enrich_onion:gitlab-issue] (optional)

[enrich_onion:gitlab-issue] (optional)
in_index = gitlab-issues_enriched
out_index = gitlab-issues-onion_enriched
data_source = gitlab-issuesmerge requestprojects.json{
    "Chaoss": {
        "gitlab:merge": [
            "https://gitlab.com/Molly/first",
            "https://gitlab.com/Molly/lab%2Fsecond"
        ],
    }
}setup.cfg[panels]
gitlab-merges = true

[gitlab:merge]
category = merge_request
raw_index = gitlab-mrs_raw
enriched_index = gitlab-mrs_enriched
api-token = xxxx
sleep-for-rate = true
no-archive = true (suggested)
studies = [enrich_onion:gitlab-merge] (optional)

[enrich_onion:gitlab-merge] (optional)
in_index = gitlab-mrs_enriched
out_index = gitlab-mrs-onion_enriched
data_source = gitlab-mergesgitter↑Messages from gitter roomsYou have to join the rooms you want to mine.projects.json{
    "Chaoss": {
        "gitter": [
            "https://gitter.im/jenkinsci/jenkins",
        ]
    }
}setup.cfg[gitter]
raw_index = gitter_raw
enriched_index = gitter_enriched_raw
api-token = xxxxx
sleep-for-rate = true
sleep-time = "300" (optional)
no-archive = true (suggested)google_hits↑Number of hits for a set of keywords from Googleprojects.json{
    "Chaoss": {
        "google_hits": [
            "bitergia grimoirelab"
        ]
    }
}setup.cfg[google_hits]
raw_index = google_hits_raw
enriched_index =google_hits_enrichedgroupsio↑Messages from GroupsioTo know the lists you are subscribed to:https://gist.github.com/valeriocos/ad33a0b9b2d13a8336230c8c59df3c55projects.json{
    "Chaoss": {
        "groupsio": [
            "group1",
            "group2"
        ]
    }
}setup.cfg[groupsio]
raw_index = groupsio_raw
enriched_index = groupsio_enriched
email = yyyy
password = xxxxhyperkitty↑Messages from a HyperKittyprojects.json{
    "Chaoss": {
        "hyperkitty": [
            "https://lists.mailman3.org/archives/list/mailman-users@mailman3.org"
        ]
    }
}setup.cfg[hyperkitty]
raw_index = hyperkitty_raw
enriched_index = hyperkitty_enrichedjenkins↑Builds from a Jenkinsprojects.json{
    "Chaoss": {
        "jenkins": [
            "https://build.opnfv.org/ci"
        ]
    }
}setup.cfg[jenkins]
raw_index = jenkins_raw
enriched_index = jenkins_enriched
no-archive = true (suggested)jira↑Issues data from JIRA issue trackersprojects.json{
    "Chaoss":{
        "jira": [
            "https://jira.opnfv.org"
        ]
    }
}setup.cfg[jira]
raw_index = jira_raw
enriched_index = jira_enriched
no-archive = true (suggested)
backend-user = yyyy (optional)
backend-password = xxxx (optional)kitsune↑Questions and answers from KitSuneprojects.json{
    "Chaoss": {
        "kitsune": [
            ""
        ]
    }
}setup.cfg[kitsune]
raw_index = kitsune_raw
enriched_index = kitsune_enrichedmattermost↑Messages from Mattermost channelsprojects.json{
    "Chaoss": {
        "mattermost": [
            "https://chat.openshift.io 8j366ft5affy3p36987pcugaoa"
        ]
    }
}setup.cfg[mattermost]
raw_index = mattermost_raw
enriched_index = mattermost_enriched
api-token = xxxx
studies = [enrich_demography:mattermost] (optional)

[enrich_demography:mattermost] (optional)mbox↑Messages from MBox filesFor mbox files, it is needed the name of the mailing list and the path where the mboxes can be found. In the example
below, the name of the mailing list is set to "mirageos-devel".projects.json{
    "Chaoss": {
        "mbox": [
            "mirageos-devel /home/bitergia/mbox/mirageos-devel/"
        ]
    }
}setup.cfg[mbox]
raw_index = mbox_raw
enriched_index = mbox_enrichedmediawiki↑Pages and revisions from MediaWiki-projects.json{
    "Chaoss": {
        "mediawiki": [
            "https://www.mediawiki.org/w https://www.mediawiki.org/wiki"
        ]
    }
}setup.cfg[mediawiki]
raw_index = mediawiki_raw
enriched_index = mediawiki_enriched
no-archive = true (suggested)meetup↑Events from Meetup groupsFor meetup groups it is only needed the identifier of the meetup group
and an API token:https://chaoss.github.io/grimoirelab-tutorial/gelk/meetup.html#gathering-meetup-groups-dataprojects.json{
    "Chaoss": {
        "meetup": [
        "Alicante-Bitergia-Users-Group",
        "South-East-Bitergia-User-Group"
        ]
    }
}setup.cfg[meetup]
raw_index = meetup_raw
enriched_index = meetup_enriched
api-token = xxxx
sleep-for-rate = true
sleep-time = "300" (optional)
no-archive = true (suggested)mozillaclub↑Events from Mozillaclubprojects.json{
    "Chaoss": {
        "mozillaclub": [
            "https://spreadsheets.google.com/feeds/cells/1QHl2bjBhMslyFzR5XXPzMLdzzx7oeSKTbgR5PM8qp64/ohaibtm/public/values?alt=json"
        ]
    }
}setup.cfg[mozillaclub]
raw_index = mozillaclub_raw
enriched_index = mozillaclub_enrichednntp↑Articles from NNTP newsgroupsThe way to setup netnews is adding the server and the news channel to be monitored. In the example below,
thenews.myproject.orgis the server name.projects.json{
    "Chaoss": {
        "nntp": [
            "news.myproject.org mozilla.dev.tech.crypto.checkins",
            "news.myproject.org mozilla.dev.tech.electrolysis",
            "news.myproject.org mozilla.dev.tech.gfx",
            "news.myproject.org mozilla.dev.tech.java"
        ]
    }
}setup.cfg[nntp]
raw_index = nntp_raw
enriched_index =  nntp_enrichedpagure↑Issues from Pagure repositoriesprojects.json{
    "Chaoss": {
        "pagure": [
            "https://pagure.io/Test-group/Project-example-namespace"
        ]
    }
}setup.cfg[pagure]
raw_index = pagure_raw
enriched_index = pagure_enriched
api-token = xxxx
sleep-for-rate = true
sleep-time = "300" (optional)
no-archive = true (suggested)phabricator↑Tasks from Phabricatorprojects.json{
    "Chaoss": {
        "phabricator": [
            "https://phabricator.wikimedia.org"
        ]
    }
}setup.cfg[phabricator]
raw_index = phabricator_raw
enriched_index = phabricator_enriched
api-token = xxxx
no-archive = true (suggested)pipermail↑Messages from Pipermailprojects.json{
    "Chaoss": {
        "pipermail": [
            "https://lists.linuxfoundation.org/pipermail/grimoirelab-discussions/"
        ]
    }
}setup.cfg[pipermail]
raw_index = pipermail_raw
enriched_index = pipermail_enrichedpuppetforge↑Modules and their releases from Puppet's forgeprojects.json{
    "Chaoss": {
        "puppetforge": [
            ""
        ]
    }
}setup.cfg[puppetforge]
raw_index = puppetforge_raw
enriched_index = puppetforge_enrichedredmine↑Issues from Redmineproject.json{
    "Chaoss": {
        "redmine": [
            "http://tracker.ceph.com/"
        ]
    }
}setup.cfg[redmine]
raw_index = redmine_raw
enriched_index = redmine_enriched
api-token = XXXXXremo↑Events, people and activities from ReMoproject.json{
    "Chaoss": {
        "remo": [
            "https://reps.mozilla.org"
        ]
    }
}setup.cfg[remo]
raw_index = remo_raw
enriched_index = remo_enrichedrocketchat↑Messages from Rocketchat channelsprojects.json{
    "Chaoss": {
        "rocketchat": [
            "https://open.rocket.chat general"
        ]
    }
}setup.cfg[rocketchat]
raw_index = rocketchat_raw
enriched_index = rocketchat_enriched
api-token = xxxx
sleep-for-rate = true
user-id = xxxx
no-archive = true (suggested)rss↑Entries from RSS feedsproject.json{
    "Chaoss": {
        "remo": [
            "https://reps.mozilla.org"
        ]
    }
}setup.cfg[rss]
raw_index = rss_raw
enriched_index = rss_enrichedslack↑Messages from Slack channelsThe information needed to monitor slack channels is the channel id.projects.json{
    "Chaoss": {
        "slack": [
            "A195YQBLL",
            "A495YQBM2"
        ]
    }
}setup.cfg[slack]
raw_index = slack_raw
enriched_index = slack_enriched
api-token = xxxx
no-archive = true (suggested)stackexchange↑Questions, answers and comments from StackExchangeprojects.json{
    "Chaoss": {
        "stackexchange": [
            "http://stackoverflow.com/questions/tagged/chef",
            "http://stackoverflow.com/questions/tagged/chefcookbook",
            "http://stackoverflow.com/questions/tagged/ohai",
            "http://stackoverflow.com/questions/tagged/test-kitchen",
            "http://stackoverflow.com/questions/tagged/knife"
        ]
    }
}setup.cfg[stackexchange]
raw_index = stackexchange_raw
enriched_index = stackexchange_enriched
api-token = xxxx
no-archive = true (suggested)supybot↑Messages from Supybot log filesFor supybot files, it is needed the name of the IRC channel and the path where the logs can be found. In the example
below, the name of the channel is set to "irc://irc.freenode.net/atomic".projects.json{
    "Chaoss": {
        "supybot": [
            "irc://irc.freenode.net/atomic /home/bitergia/irc/percevalbot/logs/ChannelLogger/freenode/#atomic"
        ]
    }
}setup.cfg[supybot]
raw_index = supybot_raw
enriched_index = supybot_enrichedtelegram↑Messages from TelegramYou need to have an API token:https://github.com/chaoss/grimoirelab-perceval#telegramprojects.json{
    "Chaoss": {
        "telegram": [
            "Mozilla_analytics"
        ]
    }
}setup.cfg[telegram]
raw_index = telegram_raw
enriched_index = telegram_enriched
api-token = XXXXXtwitter↑Messages from TwitterYou need to provide asearch queryand an API token (which requires to create anapp). The script athttps://gist.github.com/valeriocos/7d4d28f72f53fbce49f1512ba77ef5f6helps obtaining a token.projects.json{
    "Chaoss": {
        "twitter": [
            "bitergia"
        ]
    }
}setup.cfg[twitter]
raw_index = twitter_raw
enriched_index = twitter_enriched
api-token = XXXXweblate↑Changes from WeblateYou need to have an API token: The token can be obtained after registering to a weblate
instance (e.g.,https://translations.documentfoundation.org/), via the page /accounts/profile/#apiprojects.json{
    "Chaoss": {
        "weblate": [
            "https://translations.documentfoundation.org"
        ]
    }
}setup.cfg[weblate]
raw_index = weblate_raw
enriched_index = weblate_enriched
api-token = XXXX
no-archive = true (suggested)
sleep-for-rate = true (suggested)
studies = [enrich_demography:weblate] (optional)

[enrich_demography:weblate] (optional)Micro Mordred↑Micro Mordred is a simplified version of Mordred which omits the use of its scheduler. Thus, Micro Mordred allows running single Mordred tasks (e.g., raw collection, enrichment) per execution.Micro Mordred is located in thesirmordred/utilsfolder of this same repository. It can be executed via command line, its parameters are summarized below:--debug: execute Micro Mordred in debug mode

--raw: activate raw task

--enrich: activate enrich task

--identities: activate merge identities task

--panels: activate panels task

--cfg: path of the configuration file

--backends: list of cfg sections where the active tasks will be executed

--logs-dir: single parameter denoting the path of folder in which logs are to be storedExamples of possible executions are shown below:cd .../grimoirelab-sirmordred/sirmordred/utils/
micro.py --raw --enrich --cfg ./setup.cfg --backends git # execute the Raw and Enrich tasks for the Git cfg section
micro.py --panels # execute the Panels task to load the Sigils panels to Kibiter
micro.py --raw --enrich --debug --cfg ./setup.cfg --backends groupsio --logs-dir logs # execute the raw and enriched tasks for the groupsio cfg section with debug mode on and logs being saved in the folder logs in the same directory as micro.py