Package: intelmq-certbund-contact
Two expert bots to lookup contact information in a database and apply notification rulesPart of theintelmq-cb-mailgen solution.An overview of the setup can be gained from theIntelMQ Mailgen Docs.Contact DBAutomatic versus manual contactsTwo types of contacts are supported.
Both are modeled with a set of tables that differ
in the table name and in some columns.'Automatic' for externally maintained infosThe tables ending with_automaticshall hold contact
information that are maintained externally to the system.
They are designed so they can be updated or reimported
in a straight forward way. Several import sources are possible.Therefore they have columns forimport_sourceandimport_timein order to later decide which information to use. And there are
no fields for additional information as those informatiom may potentially
get lost or be incorrect if the contents of the database changes
during an update or re-import.The example for an external import source is contact data from RIPE,
please seeREADME-ripe-import.md.'Manual' to hold special casesIf the automatic contacts contain mistakes or a special case is needed,
the "manual" tables (that have no additional name suffix) can be used.
The information in the manual tables is to be considered the primary
source and maintained within this contact database.Those special information shall be entered by regular users of the system
and should not interfere with the original external data. To keep the
system robust and consistent, there are no direct links
to the automatic tables.To add special information to an already existing contact in the automatic
tables, all relevant tables entries of this specific automatic contact
have to be copied into the manual tables. The manual contact tables lack
columns about import source, but have comment and annotations
possibilities.The comments are free-text fields that can be used to record additional
information that cannot not be expressed with the existing structured fields.
This allows for rare entries or to see what additional info people
would like to see in the database in future version of the database schema.The annotations can be used as general "tags" to steer behaviour of
the system that can be configured on the level of administration rights,
but selected on the user level. The "simple" tags are designed to best be used
to denote a group of properties for sending behaviour that can be combined.
It is recommended to keep the numer of tags low.
The tags have to be tuned after gaining experience in production use
and they will be easy to use if good tags are chosen over time.Theinhibition-annotations expose a generic way to prevent
sending out information based on field contents of the intelmq event
to users. Which is important if no administrator is available on short notice
or the behaviour is an exception that does not indicate a change in the general
behaviour groups. Theinhibition-annotations are powerful
at the expense of being harder to use.How to update external sources?The manual entries have additional information to the automatic ones,
but also contain copied parts of the automatic entries.If an automatic import source is updated, the additional
and copied information may be out of date and in need of change.Example 1: the manual entry corrects a contact email address.
With the next update, the email address in the RIPE database is
updated to the correct one. Ideally the manual entry is deleted
as it is not needed anymore. (Though it does not harm if is kept
for a while.)Example 2: again the manual entry corrects a contact email address.
Now a network not is not served anymore by the same organisation coming
with the RIPE database update. In this case the CIDR has to be removed
from the manuel entry, because otherwise notifications will be send that
should not go to the organisation anymore.As there is no algorithmic way to know which info is better after an update
and thus which kind of changes have to be made,
a human has to involved.At time of writing, this has to be done by an administrator, e.g once
a week or every two weeks for each import source, like:Check diffs if you would import for necessary changes.Stop the lookup expert-bot.Do the manual changes (e.g. with fody).Completly replace the automatic tables of this source with the new version.Restart the lookup bot.For RIPE there is a script that will show the difference
between old and new automatic entries
and which manual entries are affected by theses changes (see link
to the RIPE documentation above).Database SetupRequires the python module psycopg2 (which is already installed if
you also use the postgresql-output bot).Requires at least Postgresql v>=9.5 (compare to intelmq-fody-backend).The following commands assume that PostgreSQL is running and listening on the
default port. They create a database called "contactdb" which matches the
default configuration of the bot.su - postgres

    createdb --encoding=UTF8 --template=template0 contactdb
    psql -f /usr/share/doc/intelmq-certbund-contact/sql/initdb.sql   contactdbA database user with the right to select the data in the Contact DB
must be created.  This is the account, which will be used in the bot's
configuration for accessing the database.createuser intelmq --pwprompt
    psql -c "GRANT SELECT ON ALL TABLES IN SCHEMA public TO intelmq;" contactdbAdding default email tagsThe names, values and default values for tags directly associated with
an email address have to be saved in the database.
(Maybe future version of fody-backend and fody will allow editing them, but
as of 2021-02-25 they don't.)The following example adds two tag names with a few values:INSERTINTOtag_name(tag_name,tag_name_order)VALUES('Format',1);-- -> id 1INSERTINTOtag_name(tag_name,tag_name_order)VALUES('Constituency',2);-- -> id 2COPYtag(tag_name_id,tag_value,tag_description,is_default)FROMSTDINWITHCSV;1,CSV_inline,CSVinline,true1,CSV_attachment,CSVattachment,false2,network_operators,NetworkOperators,true2,government,Government,false2,CNI,CriticalNationalInfrastructure,false2,CNI_energy,CNIEnergy,false\.(Tags for emails have been added with release 0.9.4 in 2019-05.)Schema UpdatesWhen upgrading to new version take a look atUPDATE.mg.Adding New ContactsContacts can be added to the database directly using SQL.
Though most users are expected to use a frontend application like Fody.When put into the tables for manual contacts, the entries
will take precedence over contacts which
were imported automatically, i.e. byripe_import.py.Connect to the database:su-postgrespsqlcontactdbAdd a contact:-- 1. Prepare contact information\setasn3320-- unique name of the organization:\setorg_name'org1'\setorg_comment'Example comment on organization.'\setcontact_email'test@example.com'\setcontact_comment'Test comment on contact.'-- set the notification interval in seconds-- an interval of -1 means no notifications will be generated\setnotification_interval0-- 2. Add new contactBEGINTRANSACTION;INSERTINTOautonomous_system(number)VALUES(:asn);WITHnew_orgAS(INSERTINTOorganisation(name,comment)VALUES(:'org_name',:'org_comment')RETURNINGid),new_contactAS(INSERTINTOcontact(email,format_id,comment)VALUES(:'contact_email',2,:'contact_comment')RETURNINGid),new_otaAS(INSERTINTOorganisation_to_asn(organisation_id,asn_id,notification_interval)VALUES((SELECTidfromnew_org),:asn,:notification_interval))INSERTINTOrole(organisation_id,contact_id)VALUES((SELECTidfromnew_org),(SELECTidfromnew_contact));COMMITTRANSACTION;Example change tagsIf we call the simple text annotations tags, how could we change a tag
that has already be used?Of course the notification rules may have to be changed
(see mailgen documentation) and possibly the list of known tags
that the fody-backend sends to fody.For example the following sql command on databasecontactdbchange the tagwhitelist-malwaretoWhitelist:Malware:BEGIN;UPDATEautonomous_system_annotationSETannotation='{"tag": "Whitelist:Malware"}'WHEREannotation->>'tag'='whitelist-malware';UPDATEorganisation_annotationSETannotation='{"tag": "Whitelist:Malware"}'WHEREannotation->>'tag'='whitelist-malware';UPDATEnetwork_annotationSETannotation='{"tag": "Whitelist:Malware"}'WHEREannotation->>'tag'='whitelist-malware';UPDATEfqdn_annotationSETannotation='{"tag": "Whitelist:Malware"}'WHEREannotation->>'tag'='whitelist-malware';COMMIT;Suppress notification of contacts based upon certain criteria:It ist possible to suppress the notification of contacts based upon certain
criteria. Such can be: AS-number, IP addresses, FQDN, or Organisations.To suppress notifications for such an Object, one has to create an annotation to
such an Object.Remember: It's up to theRule Expertto determine if such an annotation is
evaluated or not.Generating a graphic which visualizes the schema of the ContactDBYou can usepostgresql-autodocto do this. PG autodoc is available on both
debian and ubuntu via apt.