Repository: ssspamqe/kafka-chat
Pull request #7: Feature/multiple consumers creation
Author: login: DanisSharafiev
Created: 2025-04-27T17:18:59Z | Merged: 2025-04-27T18:57:11Z



Diff:
diff --git a/back/consumer/README.md b/back/consumer/README.md
index 849f0bc..c228271 100644
--- a/back/consumer/README.md
+++ b/back/consumer/README.md
@@ -1,39 +1,74 @@
 # Kafka Chat Consumer
 
-## How to Start the Server
+## Overview
+The Kafka Chat Consumer is a FastAPI-based microservice designed to handle WebSocket connections and consume messages from Kafka topics. It facilitates real-time communication by subscribing users to specific topics and delivering messages efficiently.
+
+---
+
+## Features
+- **WebSocket Endpoints**: Real-time communication with clients.
+- **Kafka Integration**: Subscribes to Kafka topics and processes messages.
+- **Health Check**: Ensures the service is running and operational.
 
-1. **Install Dependencies**:
+---
+
+## Prerequisites
+- Python 3.9 or higher
+- Kafka server running and accessible
+
+---
+
+## Installation
+
+1. **Clone the Repository**:
+   ```bash
+   git clone <repository-url>
+   cd kafka-chat/back/consumer
+   ```
+
+2. **Install Dependencies**:
    ```bash
    pip install -r requirements.txt
    ```
 
-2. **Run the Server**:
+---
+
+## How to Start the Server
+
+1. **Run the Server**:
    ```bash
    uvicorn main:app --host 0.0.0.0 --port 8001
    ```
 
-3. **Access the API Documentation**:
+2. **Access the API Documentation**:
    - Swagger UI: [http://localhost:8001/docs](http://localhost:8001/docs)
    - ReDoc: [http://localhost:8001/redoc](http://localhost:8001/redoc)
 
 ---
 
-## API Contract
+## API Endpoints
 
 ### WebSocket Endpoints
 
-1. **`/send-message/client/{receiver}`**
-   - **Description**: Sends a message to a specific client.
+1. **`/send-message/client/{username}`**
+   - **Description**: Sends messages to a specific client.
    - **Message Format** (JSON):
      ```json
      {
        "sender": "string",
        "text": "string",
-       "tag": "tag"
+       "tag": "tag",
+       "topic": "topic_name"
      }
      ```
 
-2. **`/health`**
+2. **`/subscribing/{username}`**
+   - **Description**: Subscribes a user to a Kafka topic.
+   - **Parameters**:
+     - `username`: The username of the client.
+     - `topic`: The Kafka topic to subscribe to.
+
+3. **`/health`**
    - **Description**: Health check endpoint to verify the consumer is running.
    - **Response**:
      ```json
@@ -44,9 +79,7 @@
 
 ---
 
-## Kafka Contract
-
-### Topics
+## Kafka Topics
 
 1. **`kafka.chat.global`**
    - **Description**: Topic for global messages.
@@ -55,7 +88,8 @@
      {
        "sender": "string",
        "text": "string",
-       "tag": "tag"
+       "tag": "tag",
+       "topic": "kafka.chat.global"
      }
      ```
 
@@ -66,6 +100,25 @@
      {
        "sender": "string",
        "text": "string",
-       "tag": "tag"
+       "tag": "tag",
+       "topic": "kafka.chat.room.{chat}"
      }
-     ```
\ No newline at end of file
+     ```
+
+---
+
+## Project Structure
+- **`main.py`**: Entry point of the application.
+- **`custom_websockets/`**: Contains WebSocket endpoint implementations.
+- **`custom_kafka/`**: Kafka consumer logic.
+- **`config/`**: Configuration files for logging and application settings.
+
+---
+
+## Contributing
+Contributions are welcome! Please fork the repository and submit a pull request.
+
+---
+
+## License
+This project is licensed under the MIT License.
\ No newline at end of file
diff --git a/back/consumer/config/config.py b/back/consumer/config/config.py
index 3c7dc6c..0a0a66e 100644
--- a/back/consumer/config/config.py
+++ b/back/consumer/config/config.py
@@ -1,12 +1,7 @@
-class Variables:
-    KAFKA_BOOTSTRAP_SERVERS = "kafka:9092"
-    KAFKA_GLOBAL_TOPIC = "kafka.chat.global"
-    KAFKA_CHAT_TOPIC_PREFIX = "kafka.chat.room"
-    KAFKA_OFFSET_RESET = "earliest"
-    KAFKA_CONSUMER_GROUP = "my_consumer_group"
-
-KAFKA_CONFIG = {
-    'bootstrap.servers': Variables.KAFKA_BOOTSTRAP_SERVERS,
-    'group.id': Variables.KAFKA_CONSUMER_GROUP,
-    'auto.offset.reset': Variables.KAFKA_OFFSET_RESET
-}
+KAFKA_BOOTSTRAP_SERVERS = "kafka:9092"
+KAFKA_GLOBAL_TOPIC = "kafka.chat.global"
+KAFKA_CHAT_TOPIC_PREFIX = "kafka.chat"
+KAFKA_OFFSET_RESET = "latest"
+KAFKA_CONSUMER_GROUP = "my_consumer_group"
+MONGODB_HOST = "utility"
+MONGODB_PORT = 8002
\ No newline at end of file
diff --git a/back/consumer/custom_kafka/kafka_consumer.py b/back/consumer/custom_kafka/kafka_consumer.py
index 7ee728b..7d4a864 100644
--- a/back/consumer/custom_kafka/kafka_consumer.py
+++ b/back/consumer/custom_kafka/kafka_consumer.py
@@ -1,60 +1,61 @@
-from confluent_kafka import Consumer, KafkaError
-from config.config import KAFKA_CONFIG as conf
+from aiokafka import AIOKafkaConsumer
+import config.config as config
+import asyncio
+from pydantic import BaseModel
 from config.logger_config import logger
+import json
+from fastapi import WebSocket, WebSocketDisconnect
+import uuid
 
-logger.debug("Kafka consumer configuration: %s", conf)
-
-def create_consumer(consumers, base_conf, user_topic):
-    logger.info(f"Creating consumer for topic {user_topic}...")
-    temp_conf = base_conf.copy()
-    consumer = Consumer(temp_conf)
-    subscribe_to_topic(consumer, user_topic)
-    logger.info("Kafka consumer created successfully.")
-    consumers[user_topic] = consumer
-    logger.info(f"Consumer for topic {user_topic} added to consumers dictionary.")
-    return consumer
-
-def subscribe_to_topic(consumer, topic):
-    logger.info(f"Subscribing to topic {topic}...")
-    consumer.subscribe([topic])
-    logger.info(f"Subscribed to topic {topic} successfully.")
-
-def get_consumer(consumers, base_conf, user_topic):
-    logger.info(f"Getting consumer for topic {user_topic}...")
-    if user_topic not in consumers:
-        logger.info(f"Consumer for topic {user_topic} not found, creating a new one...")
-        if "kafka.chat.room.global" not in consumers:
-            logger.info("Consumer for 'kafka.chat.room.global' not found, creating it...")
-            global_consumer = create_consumer(consumers, base_conf, "kafka.chat.room.global")
-            consumers["kafka.chat.room.global"] = global_consumer
-        else:
-            global_consumer = consumers["kafka.chat.room.global"]
-
-        subscribe_to_topic(global_consumer, user_topic)
-        consumers[user_topic] = global_consumer
+async def create_consumer(topic=None):
+    if topic is None:
+        topic = [config.KAFKA_GLOBAL_TOPIC]
+    logger.info("Creating Kafka consumer...")
+    unique_group_id = str(uuid.uuid4())
+    consumer = AIOKafkaConsumer(
+    *topic,
+    bootstrap_servers=config.KAFKA_BOOTSTRAP_SERVERS,
+    group_id=unique_group_id,
+    auto_offset_reset=config.KAFKA_OFFSET_RESET,  
+    enable_auto_commit=True)
+    if not consumer:
+        logger.error("Failed to create Kafka consumer.")
     else:
-        consumer = consumers[user_topic]
-    return consumers[user_topic]
+        logger.info("Kafka consumer created successfully.")
+    return consumer
 
-def close_consumer(consumer):
-    logger.info("Closing consumer...")
-    consumer.close()
+async def subscribe_to_topic(consumer, topic):  
+    current_subscription = consumer.subscription()
+    
+    if topic in current_subscription:
+        logger.info(f"Already subscribed to topic: {topic}")
+        return
+    
+    new_subscription = list(current_subscription) + [topic]
+    consumer.subscribe(new_subscription)
+    logger.info(f"Subscribed to topic: {topic}. Current topics: {new_subscription}")
 
-async def consume_messages(consumer):
+async def subscribe_to_chat(consumer, chat):
+    await subscribe_to_topic(consumer, f'{config.KAFKA_CHAT_TOPIC_PREFIX}.{chat}')
+    logger.info(f"Subscribed to chat topic: {chat}")
+    
+async def consume_messages(consumer, websocket):
     try:
-        msg = consumer.poll(1.0)
-        if msg is None:
-            return None
-        if msg.error():
-            if msg.error().code() == KafkaError._PARTITION_EOF:
-                logger.info(f"End of partition reached: {msg.topic()} [{msg.partition()}] at offset {msg.offset()}")
-            else:
-                logger.info(f"Error occurred: {msg.error()}")
-            return None
-        else:
-            message = msg.value().decode('utf-8')
-            logger.info(f"Received message from topic {msg.topic()}: {message}")
-            return message
-                
-    except KeyboardInterrupt:
-        logger.info("Consumer interrupted by user.")
+        await consumer.start()
+        async for message in consumer:
+            topic = message.topic
+            structured_message = json.loads(message.value.decode('utf-8'))
+            structured_message["chat"] = topic[len(config.KAFKA_CHAT_TOPIC_PREFIX) + 1:]
+            logger.info(f"Received message: {structured_message}")
+            try:
+                await websocket.send_json(structured_message)
+                logger.info(f"Sent structured message to WebSocket: {structured_message}")
+            except WebSocketDisconnect:
+                logger.info(f"WebSocket disconnected for topic: {topic}")
+    except asyncio.CancelledError:
+        logger.warning("Message consumption cancelled.")
+    except Exception as e:
+        logger.error(f"Error consuming messages: {e}")
+    finally:
+        await consumer.stop()
+        logger.info("Kafka consumer stopped.")
\ No newline at end of file
diff --git a/back/consumer/custom_websockets/endpoints/send_client_message.py b/back/consumer/custom_websockets/endpoints/send_client_message.py
index 110e176..8f1d9e6 100644
--- a/back/consumer/custom_websockets/endpoints/send_client_message.py
+++ b/back/consumer/custom_websockets/endpoints/send_client_message.py
@@ -1,8 +1,8 @@
-from fastapi import APIRouter, WebSocket, WebSocketDisconnect
+from fastapi import APIRouter, WebSocket
+from pydantic import BaseModel
 from config.logger_config import logger
-from custom_kafka.kafka_consumer import consume_messages, get_consumer
-from config.config import KAFKA_CONFIG
-from config.config import Variables
+from custom_kafka.kafka_consumer import consume_messages, create_consumer
+import config.config as config
 
 router = APIRouter()
 state = None
@@ -11,29 +11,28 @@ def initialize_state(app_state):
     global state
     state = app_state
 
-@router.websocket("/send-message/client/{chat}")
-async def send_person_message(websocket: WebSocket, chat: str):
+@router.websocket("/send-messages/user/{username}")
+async def send_person_message(websocket: WebSocket, username: str):
+    logger.info(f"WebSocket connection request for user: {username}")
     await websocket.accept()
-    logger.info(f"WebSocket connection established with topic: {chat}")
-
-    if state:
-        topic = f'{Variables.KAFKA_CHAT_TOPIC_PREFIX}.{chat}' if chat != 'global' else Variables.KAFKA_GLOBAL_TOPIC
-        consumer = state.kafka_consumers.get(topic)
-        if not consumer:
-            logger.info(f"No consumer found for {topic}, creating one...")
-            consumer = get_consumer(state.kafka_consumers, KAFKA_CONFIG, topic)
-            state.kafka_consumers[topic] = consumer
-
-    try:
-        while True:
-            if consumer:
-                message = await consume_messages(consumer)
-                if message:
-                    logger.info(f"Sending message to topic {topic}: {message}")
-                    await websocket.send_text(message)
-    except WebSocketDisconnect:
-        logger.info(f"Client {topic} disconnected.")
-    except Exception as e:
-        logger.error(f"Error in WebSocket connection with {topic}: {e}")
-    finally:
-        logger.info(f"Closing WebSocket connection with {topic}.")
+    logger.info(f"WebSocket connection established with user: {username}")
+
+    # Здесь надо будет добавить подгрузку topics из mongodb
+    # topics = requests.get(f'http://{config.MONGODB_HOST}:{config.MONGODB_PORT}/user/{username}')
+
+    topics = []
+
+    # MOCK, нижнее удалить, когда будет подгрузка из mongodb
+    consumer = await create_consumer([f'{config.KAFKA_CHAT_TOPIC_PREFIX}.{username}', f'{config.KAFKA_GLOBAL_TOPIC}'])
+    #consumer = await create_consumer(topics) # Условно потом будет это, когда подгрузим
+
+    state.consumers[username] = consumer
+
+    await consume_messages(consumer, websocket)
+
+    state.consumers.pop(username, None)
+
+    logger.info(f"WebSocket connection closed for user: {username}")
+    await websocket.close()
+
+    
diff --git a/back/consumer/custom_websockets/endpoints/subscribe_topic.py b/back/consumer/custom_websockets/endpoints/subscribe_topic.py
new file mode 100644
index 0000000..7f9b491
--- /dev/null
+++ b/back/consumer/custom_websockets/endpoints/subscribe_topic.py
@@ -0,0 +1,28 @@
+from fastapi import APIRouter, WebSocket
+from pydantic import BaseModel
+from config.logger_config import logger
+from custom_kafka.kafka_consumer import subscribe_to_chat
+
+router = APIRouter()
+state = None
+
+class Message(BaseModel):
+    user: str
+    chats: list[str]
+    message: str
+
+def initialize_state(app_state):
+    global state
+    state = app_state
+
+@router.post("/subscribing/{username}")
+async def subscribe_user(topic: str, username: str):
+    logger.info(f"WebSocket connection request for user: {username}")
+    logger.info(f"Subscribing user {username} to topic: {topic}")
+    if username not in state.consumers:
+        logger.warning(f"No consumer found for user: {username}")
+    else:
+        await subscribe_to_chat(state.consumers[username], topic)
+        logger.info(f"Subscribed user {username} to topic: {topic}")
+
+    return {"message": "nonono mr fish you dont want to go to this bucket"} 
\ No newline at end of file
diff --git a/back/consumer/main.py b/back/consumer/main.py
index 55be6d4..b6c2700 100644
--- a/back/consumer/main.py
+++ b/back/consumer/main.py
@@ -1,6 +1,8 @@
 from fastapi import FastAPI
 from custom_websockets.endpoints.send_client_message import router as client_router
-from custom_websockets.endpoints.send_client_message import initialize_state
+from custom_websockets.endpoints.subscribe_topic import router as subscribe_router 
+from custom_websockets.endpoints.send_client_message import initialize_state as client_init_state
+from custom_websockets.endpoints.subscribe_topic import initialize_state as subscribe_init_state
 import asyncio
 from contextlib import asynccontextmanager
 from config.logger_config import logger
@@ -9,10 +11,11 @@
 async def lifespan(app: FastAPI):
     logger.info("FastAPI application startup event triggered.")
     logger.info("Creating temporary state for FastAPI application.")
-    app.state.kafka_consumers = {}  # topic -> consumer
-    app.state.background_tasks = []
 
-    initialize_state(app.state)
+    client_init_state(app.state)
+    subscribe_init_state(app.state)
+
+    app.state.consumers = {}
 
     try:
         yield
@@ -21,20 +24,10 @@ async def lifespan(app: FastAPI):
     finally:
         logger.info("FastAPI application shutdown event triggered.")
 
-        for task in app.state.background_tasks:
-            task.cancel()
-            try:
-                await task
-            except asyncio.CancelledError:
-                logger.info("Background task cancelled successfully.")
-
-        for consumer in app.state.kafka_consumers.values():
-            consumer.close()
-        logger.info("Kafka consumers closed.")
-
 app = FastAPI(lifespan=lifespan)
 
 app.include_router(client_router)
+app.include_router(subscribe_router)
 
 @app.get("/health")
 async def health_check():
diff --git a/back/consumer/requirements.txt b/back/consumer/requirements.txt
index ae4fafb..ac2559b 100644
--- a/back/consumer/requirements.txt
+++ b/back/consumer/requirements.txt
@@ -1,6 +1,8 @@
 fastapi==0.95.2
-confluent-kafka==1.9.2
 aiokafka==0.8.0
 uvicorn==0.22.0
 asyncio==3.4.3
-websockets==11.0.3
\ No newline at end of file
+websockets==11.0.3
+kafka-python==2.0.2
+pydantic==1.10.7
+requests==2.28.1
\ No newline at end of file
diff --git a/back/producer/config/config.py b/back/producer/config/config.py
index cb5de51..84e704e 100644
--- a/back/producer/config/config.py
+++ b/back/producer/config/config.py
@@ -1,4 +1,4 @@
 class Variables:
     KAFKA_BOOTSTRAP_SERVERS = "kafka:9092"
     KAFKA_GLOBAL_TOPIC = "kafka.chat.global"
-    KAFKA_CHAT_TOPIC_PREFIX = "kafka.chat.room"
+    KAFKA_CHAT_TOPIC_PREFIX = "kafka.chat"
diff --git a/back/producer/custom_kafka/kafka_producer.py b/back/producer/custom_kafka/kafka_producer.py
index ac29d15..f9fb821 100644
--- a/back/producer/custom_kafka/kafka_producer.py
+++ b/back/producer/custom_kafka/kafka_producer.py
@@ -28,7 +28,7 @@ def send_message_to_global(message: Message):
 
 def send_message_to_chat(chat, message: Message):
     topic_name = f'{config.Variables.KAFKA_CHAT_TOPIC_PREFIX}.{chat}'
-    create_kafka_topic_if_not_exists(f"kafka.chat.room.{chat}")
+    create_kafka_topic_if_not_exists(f"kafka.chat.{chat}")
     serialized_message = json.dumps(message.__dict__).encode('utf-8')
 
     logger.info(f"Sending message to chat {chat} in topic {topic_name}: {message.sender} - {message.text}, tag: {message.tag}")
diff --git a/back/producer/custom_websockets/endpoints/send_global_message.py b/back/producer/custom_websockets/endpoints/send_global_message.py
deleted file mode 100644
index 9628af8..0000000
--- a/back/producer/custom_websockets/endpoints/send_global_message.py
+++ /dev/null
@@ -1,37 +0,0 @@
-from fastapi import APIRouter, WebSocket, WebSocketDisconnect
-from custom_kafka import kafka_producer
-import json
-import logging
-
-router = APIRouter()
-logger = logging.getLogger("kafka_chat_logger")
-
-@router.websocket("/send-message/global")
-async def send_message_to_global(websocket:WebSocket):
-    await websocket.accept()
-    logger.info("/send-message/global connected")
-    try:
-        while True:
-            raw_send_message_json = await websocket.receive_text()
-            logger.info(f"Received raw message: {raw_send_message_json}")
-            
-            send_message_request = parse_send_global_message_request(raw_send_message_json)
-            
-            message = kafka_producer.Message(sender=send_message_request.sender, text=send_message_request.text, tag=send_message_request.tag)
-            
-            kafka_producer.send_message_to_global(message)
-            
-    except WebSocketDisconnect:
-        logger.info(f"{websocket.user} disconnected")
-
-
-def parse_send_global_message_request(raw_data: str):
-    data = json.loads(raw_data)
-    return SendGlobalMessageRequest(sender=data["sender"], text=data["text"], tag = data["tag"])
-    
-    
-class SendGlobalMessageRequest:
-    def __init__(self, sender: str, text: str, tag:str):
-        self.text = text
-        self.sender = sender
-        self.tag = tag
diff --git a/back/producer/main.py b/back/producer/main.py
index 7ba05e8..08944b4 100644
--- a/back/producer/main.py
+++ b/back/producer/main.py
@@ -1,8 +1,6 @@
 from fastapi import FastAPI
 from custom_websockets.endpoints.send_chat_message import router as chat_router
-from custom_websockets.endpoints.send_global_message import router as global_router
 
 app = FastAPI()
 
 app.include_router(chat_router)
-app.include_router(global_router)
\ No newline at end of file
diff --git a/docker-compose.yml b/docker-compose.yml
index c797762..0e16c54 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -18,10 +18,11 @@ services:
     environment:
       KAFKA_BROKER_ID: 1
       KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
+      ALLOW_PLAINTEXT_LISTENER: yes
       KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
-      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
       KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
       KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
+      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
     ports:
       - "29092:29092"
       - "9092:9092"


Reviews:
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This PR introduces multiple consumers creation support by updating Kafka configurations, refactoring consumer state management, and modifying message endpoints. Key changes include:
- Updating Kafka topic prefixes and consumer configurations (including switching to aiokafka).
- Removing the global messaging endpoint in the producer.
- Adding a new subscription endpoint and modifying WebSocket handling in the consumer.

### Reviewed Changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated 2 comments.

<details>
<summary>Show a summary per file</summary>

| File                                                                 | Description                                                                     |
|----------------------------------------------------------------------|---------------------------------------------------------------------------------|
| docker-compose.yml                                                   | Updated Kafka environment variables and listener configurations.              |
| back/producer/main.py                                                | Removed the global message router, aligning with the removal of global messaging.|
| back/producer/custom_websockets/endpoints/send_global_message.py       | Removed the unused global messaging endpoint.                                 |
| back/producer/custom_kafka/kafka_producer.py                           | Updated the chat topic prefix for consistency with configuration changes.       |
| back/producer/config/config.py                                         | Adjusted topic prefix to match new messaging topics.                          |
| back/consumer/main.py                                                 | Revised state management by replacing kafka_consumers with consumers.           |
| back/consumer/custom_websockets/endpoints/send_client_message.py        | Added a new endpoint for subscribing users and updated WebSocket handling.      |
| back/consumer/custom_kafka/kafka_consumer.py                           | Switched to aiokafka, modified subscription logic, and enhanced message consumption. |
| back/consumer/config/config.py                                         | Updated Kafka consumer configuration and added MongoDB connection details.      |
| back/consumer/README.md                                                | Updated endpoint documentation to reflect new routes and message formats.       |
</details>


<details>
<summary>Files not reviewed (1)</summary>

* **back/consumer/requirements.txt**: Language not supported
</details>



state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-27T17:19:50Z
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This PR introduces support for multiple consumers by updating Kafka topic naming conventions, reworking WebSocket endpoints, and refactoring consumer configurations. Key changes include:
- Removal of the global message WebSocket endpoint from the producer and corresponding code cleanup.
- Updates to Kafka topic prefixes and configuration across both producer and consumer services.
- Addition of a new subscription endpoint for users and refactoring of consumer logic using aiokafka.

### Reviewed Changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 2 comments.

<details>
<summary>Show a summary per file</summary>

| File                                                                 | Description                                                                  |
|----------------------------------------------------------------------|------------------------------------------------------------------------------|
| docker-compose.yml                                                   | Added an environment variable and re-ordered listener configuration.         |
| back/producer/main.py                                                | Removed the global message endpoint routing for the producer.                |
| back/producer/custom_websockets/endpoints/send_global_message.py       | Removed the deprecated global messaging endpoint file.                     |
| back/producer/custom_kafka/kafka_producer.py                           | Updated topic naming to align with the new prefix.                           |
| back/producer/config/config.py                                         | Updated Kafka topic prefix to the new pattern.                               |
| back/consumer/main.py                                                 | Reworked WebSocket endpoints and state initialization for consumers.         |
| back/consumer/custom_websockets/endpoints/subscribe_topic.py            | Added a new subscription endpoint for user topic subscriptions.              |
| back/consumer/custom_websockets/endpoints/send_client_message.py        | Updated WebSocket route and consumer creation logic for client messaging.   |
| back/consumer/custom_kafka/kafka_consumer.py                           | Switched from confluent_kafka to aiokafka and updated subscription handling.   |
| back/consumer/config/config.py                                         | Adjusted consumer configuration (offset reset and topic prefix) and added MongoDB settings.     |
| back/consumer/README.md                                                | Updated documentation to reflect the new endpoints and configuration.        |
</details>


<details>
<summary>Files not reviewed (1)</summary>

* **back/consumer/requirements.txt**: Language not supported
</details>

<details>
<summary>Comments suppressed due to low confidence (1)</summary>

**back/consumer/main.py:27**
* The removal of background task cancellation and Kafka consumer closing logic may lead to resource leaks on shutdown; please ensure proper cleanup is implemented.
```
-        for task in app.state.background_tasks:
```
</details>


state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-27T17:52:06Z

Review comments:
diff_hunk: @@ -1,39 +1,56 @@
-from fastapi import APIRouter, WebSocket, WebSocketDisconnect
+from fastapi import APIRouter, WebSocket
+from pydantic import BaseModel
 from config.logger_config import logger
-from custom_kafka.kafka_consumer import consume_messages, get_consumer
-from config.config import KAFKA_CONFIG
-from config.config import Variables
+from custom_kafka.kafka_consumer import consume_messages, create_consumer, subscribe_to_chat
+import config.config as config
+import requests
 
 router = APIRouter()
 state = None
 
+class Message(BaseModel):
+    user: str
+    chats: list[str]
+    message: str
+
 def initialize_state(app_state):
     global state
     state = app_state
 
-@router.websocket("/send-message/client/{chat}")
-async def send_person_message(websocket: WebSocket, chat: str):
+@router.post("/subsribing/{username}")
path: back/consumer/custom_websockets/endpoints/send_client_message.py
user: login: Copilot
body: The route path '/subsribing/{username}' contains a typo. Consider renaming it to '/subscribing/{username}'.
```suggestion
@router.post("/subscribing/{username}")
```
created_at: 2025-04-27T17:19:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,60 +1,59 @@
-from confluent_kafka import Consumer, KafkaError
-from config.config import KAFKA_CONFIG as conf
+from aiokafka import AIOKafkaConsumer
+import config.config as config
+import asyncio
+from pydantic import BaseModel
 from config.logger_config import logger
+import json
+from fastapi import WebSocket, WebSocketDisconnect
+import uuid
 
-logger.debug("Kafka consumer configuration: %s", conf)
-
-def create_consumer(consumers, base_conf, user_topic):
-    logger.info(f"Creating consumer for topic {user_topic}...")
-    temp_conf = base_conf.copy()
-    consumer = Consumer(temp_conf)
-    subscribe_to_topic(consumer, user_topic)
-    logger.info("Kafka consumer created successfully.")
-    consumers[user_topic] = consumer
-    logger.info(f"Consumer for topic {user_topic} added to consumers dictionary.")
-    return consumer
-
-def subscribe_to_topic(consumer, topic):
-    logger.info(f"Subscribing to topic {topic}...")
-    consumer.subscribe([topic])
-    logger.info(f"Subscribed to topic {topic} successfully.")
-
-def get_consumer(consumers, base_conf, user_topic):
-    logger.info(f"Getting consumer for topic {user_topic}...")
-    if user_topic not in consumers:
-        logger.info(f"Consumer for topic {user_topic} not found, creating a new one...")
-        if "kafka.chat.room.global" not in consumers:
-            logger.info("Consumer for 'kafka.chat.room.global' not found, creating it...")
-            global_consumer = create_consumer(consumers, base_conf, "kafka.chat.room.global")
-            consumers["kafka.chat.room.global"] = global_consumer
-        else:
-            global_consumer = consumers["kafka.chat.room.global"]
-
-        subscribe_to_topic(global_consumer, user_topic)
-        consumers[user_topic] = global_consumer
+async def create_consumer(topic = [config.KAFKA_GLOBAL_TOPIC]):
+    logger.info("Creating Kafka consumer...")
+    unique_group_id = str(uuid.uuid4())
+    consumer = AIOKafkaConsumer(
+    *topic,
+    bootstrap_servers=config.KAFKA_BOOTSTRAP_SERVERS,
+    group_id=unique_group_id,
+    auto_offset_reset=config.KAFKA_OFFSET_RESET,  
+    enable_auto_commit=True)
+    if not consumer:
+        logger.error("Failed to create Kafka consumer.")
     else:
-        consumer = consumers[user_topic]
-    return consumers[user_topic]
+        logger.info("Kafka consumer created successfully.")
+    return consumer
 
-def close_consumer(consumer):
-    logger.info("Closing consumer...")
-    consumer.close()
+async def subscribe_to_topic(consumer, topic):
+    current_subscription = consumer.subscription()
+    
+    if topic in current_subscription:
+        logger.info(f"Already subscribed to topic: {topic}")
+        return
path: back/consumer/custom_kafka/kafka_consumer.py
user: login: Copilot
body: Returning early when the topic is already in the current subscription may lead to inconsistent behavior. Consider either always returning the consumer or documenting the early return behavior to ensure consistent usage.
created_at: 2025-04-27T17:19:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,28 @@
+from fastapi import APIRouter, WebSocket
+from pydantic import BaseModel
+from config.logger_config import logger
+from custom_kafka.kafka_consumer import subscribe_to_chat
+
+router = APIRouter()
+state = None
+
+class Message(BaseModel):
+    user: str
+    chats: list[str]
+    message: str
+
+def initialize_state(app_state):
+    global state
+    state = app_state
+
+@router.post("/subscribing/{username}")
+async def subscribe_user(topic: str, username: str):
+    logger.info(f"WebSocket connection request for user: {username}")
+    logger.info(f"Subscribing user {username} to topic: {topic}")
+    if username not in state.consumers:
+        logger.warning(f"No consumer found for user: {username}")
+    else:
+        await subscribe_to_chat(state.consumers[username], topic)
+        logger.info(f"Subscribed user {username} to topic: {topic}")
+
+    return {"message": "nonono mr fish you dont want to go to this bucket"} 
path: back/consumer/custom_websockets/endpoints/subscribe_topic.py
user: login: Copilot
body: [nitpick] The response message contains informal language that may not be appropriate for production; consider updating it to a more professional tone.
```suggestion
    return {"message": "The user has been successfully subscribed to the topic."}
```
created_at: 2025-04-27T17:52:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,60 +1,59 @@
-from confluent_kafka import Consumer, KafkaError
-from config.config import KAFKA_CONFIG as conf
+from aiokafka import AIOKafkaConsumer
+import config.config as config
+import asyncio
+from pydantic import BaseModel
 from config.logger_config import logger
+import json
+from fastapi import WebSocket, WebSocketDisconnect
+import uuid
 
-logger.debug("Kafka consumer configuration: %s", conf)
-
-def create_consumer(consumers, base_conf, user_topic):
-    logger.info(f"Creating consumer for topic {user_topic}...")
-    temp_conf = base_conf.copy()
-    consumer = Consumer(temp_conf)
-    subscribe_to_topic(consumer, user_topic)
-    logger.info("Kafka consumer created successfully.")
-    consumers[user_topic] = consumer
-    logger.info(f"Consumer for topic {user_topic} added to consumers dictionary.")
-    return consumer
-
-def subscribe_to_topic(consumer, topic):
-    logger.info(f"Subscribing to topic {topic}...")
-    consumer.subscribe([topic])
-    logger.info(f"Subscribed to topic {topic} successfully.")
-
-def get_consumer(consumers, base_conf, user_topic):
-    logger.info(f"Getting consumer for topic {user_topic}...")
-    if user_topic not in consumers:
-        logger.info(f"Consumer for topic {user_topic} not found, creating a new one...")
-        if "kafka.chat.room.global" not in consumers:
-            logger.info("Consumer for 'kafka.chat.room.global' not found, creating it...")
-            global_consumer = create_consumer(consumers, base_conf, "kafka.chat.room.global")
-            consumers["kafka.chat.room.global"] = global_consumer
-        else:
-            global_consumer = consumers["kafka.chat.room.global"]
-
-        subscribe_to_topic(global_consumer, user_topic)
-        consumers[user_topic] = global_consumer
+async def create_consumer(topic = [config.KAFKA_GLOBAL_TOPIC]):
path: back/consumer/custom_kafka/kafka_consumer.py
user: login: Copilot
body: Using a mutable default argument (a list) can lead to unexpected behavior; consider setting the default to None and assigning the list inside the function.
```suggestion
async def create_consumer(topic=None):
    if topic is None:
        topic = [config.KAFKA_GLOBAL_TOPIC]
```
created_at: 2025-04-27T17:52:06Z
author_association: CONTRIBUTOR

PR comments:
user: login: ssspamqe
created_at: 2025-04-27T18:26:09Z
body: та хуйня боже удали себе питон с компа
author_association: OWNER
user: login: ssspamqe
created_at: 2025-04-27T18:26:17Z
body: и не позорься

author_association: OWNER