Package: domination
DominationReal-time application made to monitor and dominate Humans.From the rating of every human (from 1 to 100) sent to the kafka topicdominate,
we detect, in real time, which type they are:Shaif its rating is a multiple of 3Dowif its rating is a multiple of 5ShaDowif its rating is a multiple of 3 and 5Incompatibleif none of the above.Using a new kafka topicshadow, we make the result available to a clickhouse
table namedshadow.System Design+----------------+               +-------------+            +------------------+
|  domination    |               |  dominate   |            | domination       |
+----------------+               +-------------+            +------------------+
| python         | +-----------> |             | +--------> | python           |
| Faust producer |               | Kafka topic |            | Faust agent      |
| HumanRatings   |               |             |            | HumanCategorized |
+----------------+               +-------------+            +------------------+
                                                                      +
      +---------------------------------------------------------------+
      v
+----------------+           +-------------------+           +-------------------+
|  shadow        |           | shadow_stream     |           |  shadow_consumer  |
+----------------+           +-------------------+           +-------------------+
|                | +------>  | clickhouse table  | +------>  | clickhouse table  |
|  Kafka topic   |           | encapsulate topic |           | materialized view |
+----------------+           +-------------------+           +-------------------+
                                                                      +
      +---------------------------------------------------------------+
      v
+-------------------+
|  shadow           |
+-------------------+
| clickhouse table  |
| store rows        |
+-------------------+Structure of Kafka messages:topicdominate:{"rating": <integer>, "unique_id": "<string>"}topicshadow:{"type": <integer>, "unique_id": "<string>", "emit_timestamp": <datetime>}UsageRequirementsPython >= 3.6docker-composeStart dominationgit clone https://github.com/AzemaBaptiste/domination.git && cd domination
virtualenv -p python3.8 venv && source venv/bin/activate
pip install domination
docker-compose up -d
./create_clickhouse_tables.sh

domination worker -l infoExplore data in a new terminaldocker exec -it clickhouse bin/bash -c "clickhouse-client --multiline"
   > SELECT COUNT(type) AS COUNT, type FROM shadow 
     GROUP BY type ORDER BY (COUNT) DESC LIMIT 10;Stop dominationCtrl + C (stop faust app)
docker-compose downRun The AlgorithmYou can also run The Algorithm as a standalone. It will print the type
of every human rated from 1 to 1337.python the_algorithm.pyDevelopment# Install
virtualenv -p python3.8 venv
source venv/bin/activate
pip install -r requirements.txt
make install

# Build
make test # coverage tests
make linter # runs pylint
make buildExplore dataOpen CLI of the clickhouse clientdocker exec -it clickhouse bin/bash -c "clickhouse-client --multiline"Description of clickhouse tablesshadow_stream: consumer of the kafka topicCREATE TABLE IF NOT EXISTS shadow_stream
(
    `type` String,
    `unique_id` String,
    `emit_timestamp` DateTime
) ENGINE = Kafka()
  SETTINGS
    kafka_broker_list = 'kafka:29092',
    kafka_topic_list = 'shadow',
    kafka_group_name = 'shadow-group',
    kafka_format = 'JSONEachRow',
    kafka_skip_broken_messages = 1;shadow: final tableCREATE TABLE shadow as shadow_stream
ENGINE = MergeTree()
PARTITION BY toYYYYMM(emit_timestamp)
ORDER BY type;shadow_consumer: materialized view feeding the final table with the consumer tableCREATE MATERIALIZED VIEW shadow_consumer 
TO shadow
AS SELECT * FROM shadow_stream;Referencesblog.streamthoughts.frmedium.comTODOdeploy package to pypi using github actions