Repository: certego/BuffaLogs
Pull request #227: [Feature] Splunk Ingestion
Author: login: drona-gyawali
Created: 2025-04-03T16:11:15Z | Merged: 2025-04-10T10:39:48Z

ref: #220 

In this PR, I implemented the test case for Splunk ingestion, developed the ingestion code, and provided detailed documentation to ensure clarity and ease of use.


Diff:
diff --git a/buffalogs/impossible_travel/ingestion/base_ingestion.py b/buffalogs/impossible_travel/ingestion/base_ingestion.py
index d7e75ea3..913ea308 100644
--- a/buffalogs/impossible_travel/ingestion/base_ingestion.py
+++ b/buffalogs/impossible_travel/ingestion/base_ingestion.py
@@ -15,10 +15,12 @@ class SupportedIngestionSources(Enum):
         """Types of possible data ingestion sources
 
         * ELASTICSEARCH: The login data is extracted from Elasticsearch
+        * SPLUNK: The login data is extracted from Splunk
         * OPENSEARCH: The login data is extracted from Opensearch
         """
 
         ELASTICSEARCH = "elasticsearch"
+        SPLUNK = "splunk"
         OPENSEARCH = "opensearch"
 
     def __init__(self, ingestion_config, mapping):
diff --git a/buffalogs/impossible_travel/ingestion/ingestion_factory.py b/buffalogs/impossible_travel/ingestion/ingestion_factory.py
index 31afff9f..cc9cfc57 100644
--- a/buffalogs/impossible_travel/ingestion/ingestion_factory.py
+++ b/buffalogs/impossible_travel/ingestion/ingestion_factory.py
@@ -5,6 +5,7 @@
 from impossible_travel.ingestion.base_ingestion import BaseIngestion
 from impossible_travel.ingestion.elasticsearch_ingestion import ElasticsearchIngestion
 from impossible_travel.ingestion.opensearch_ingestion import OpensearchIngestion
+from impossible_travel.ingestion.splunk_ingestion import SplunkIngestion
 
 
 class IngestionFactory:
@@ -43,5 +44,7 @@ def get_ingestion_class(self):
                 return ElasticsearchIngestion(self.ingestion_config, self.mapping)
             case BaseIngestion.SupportedIngestionSources.OPENSEARCH:
                 return OpensearchIngestion(self.ingestion_config)
+            case BaseIngestion.SupportedIngestionSources.SPLUNK:
+                return SplunkIngestion(self.ingestion_config, self.mapping)
             case _:
                 raise ValueError(f"Unsupported ingestion source: {self.active_ingestion}")
diff --git a/buffalogs/impossible_travel/ingestion/splunk_ingestion.py b/buffalogs/impossible_travel/ingestion/splunk_ingestion.py
new file mode 100644
index 00000000..7a01a885
--- /dev/null
+++ b/buffalogs/impossible_travel/ingestion/splunk_ingestion.py
@@ -0,0 +1,138 @@
+import logging
+from datetime import datetime
+
+import splunklib.client as client
+import splunklib.results as results
+from impossible_travel.ingestion.base_ingestion import BaseIngestion
+
+
+class SplunkIngestion(BaseIngestion):
+    """
+    Concrete implementation of the BaseIngestion class for Splunk ingestion source
+    """
+
+    def __init__(self, ingestion_config: dict, mapping: dict):
+        """
+        Constructor for the Splunk Ingestion object
+        """
+        super().__init__(ingestion_config, mapping)
+        try:
+            # Create the Splunk host connection
+            self.service = client.connect(
+                host=self.ingestion_config.get("host", "localhost"),
+                port=self.ingestion_config.get("port", 8089),
+                username=self.ingestion_config.get("username"),
+                password=self.ingestion_config.get("password"),
+                scheme=self.ingestion_config.get("scheme", "http"),
+            )
+        except ConnectionError as e:
+            logging.error("Failed to establish a connection: %s", e)
+            self.service = None
+
+        self.logger = logging.getLogger(f"{__name__}.{self.__class__.__name__}")
+
+    def process_users(self, start_date: datetime, end_date: datetime) -> list:
+        """
+        Concrete implementation of the BaseIngestion.process_users abstract method
+
+        :param start_date: Initial datetime from which users are considered
+        :param end_date: Final datetime within which users are considered
+        :return: List of user strings that logged in Splunk
+        """
+        self.logger.info(f"Starting at: {start_date} Finishing at: {end_date}")
+        users_list = []
+
+        # Format dates for Splunk query
+        start_date_str = start_date.strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z"
+        end_date_str = end_date.strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z"
+
+        query = f"""
+            search index={self.ingestion_config["indexes"]}
+            earliest="{start_date_str}" latest="{end_date_str}"
+            event.category="authentication" event.outcome="success" event.type="start"
+            | stats count by user.name
+            | where isnotnull(user.name) AND user.name!=""
+        """
+
+        try:
+            search_kwargs = {
+                "earliest_time": start_date_str,
+                "latest_time": end_date_str,
+                "exec_mode": "normal",
+                "count": self.ingestion_config.get("bucket_size", 10000),
+            }
+
+            search_job = self.service.jobs.create(query, **search_kwargs)
+
+            while not search_job.is_done():
+                search_job.refresh()
+
+            results_reader = results.ResultsReader(search_job.results())
+            for result in results_reader:
+                if isinstance(result, dict) and "user.name" in result:
+                    users_list.append(result["user.name"])
+
+            self.logger.info(f"Successfully got {len(users_list)} users")
+
+        except ConnectionError:
+            self.logger.error(f"Failed to establish a connection with host: {self.ingestion_config.get('host')}")
+        except TimeoutError:
+            self.logger.error(f"Timeout reached for the host: {self.ingestion_config.get('host')}")
+        except Exception as e:
+            self.logger.error(f"Exception while querying Splunk: {e}")
+        return users_list
+
+    def process_user_logins(self, start_date: datetime, end_date: datetime, username: str) -> list:
+        """
+        Concrete implementation of the BaseIngestion.process_user_logins abstract method
+
+        :param username: Username of the user that logged in
+        :param start_date: Initial datetime from which logins are considered
+        :param end_date: Final datetime within which logins are considered
+        :return: List of login dictionaries for the specified user
+        """
+        response = []
+
+        # Format dates for Splunk query
+        start_date_str = start_date.strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z"
+        end_date_str = end_date.strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z"
+
+        # Build Splunk query to get login events for the specific user
+        query = f"""
+            search index={self.ingestion_config["indexes"]}
+            earliest="{start_date_str}" latest="{end_date_str}"
+            user.name="{username}" event.category="authentication" event.outcome="success" event.type="start"
+            | where isnotnull(source.ip)
+            | fields user.name, _time AS "@timestamp", source.geo.location.lat, source.geo.location.lon,
+              source.geo.country_name, source.as.organization.name, user_agent.original, index, source.ip, _id,
+              source.intelligence_category
+            | sort 0 @timestamp
+        """
+        try:
+            search_kwargs = {
+                "earliest_time": start_date_str,
+                "latest_time": end_date_str,
+                "exec_mode": "normal",
+                "count": self.ingestion_config.get("bucket_size", 10000),
+            }
+
+            search_job = self.service.jobs.create(query, **search_kwargs)
+
+            # Wait for the job to complete
+            while not search_job.is_done():
+                search_job.refresh()
+
+            results_reader = results.ResultsReader(search_job.results())
+            for result in results_reader:
+                if isinstance(result, dict):
+                    response.append(result)
+
+            self.logger.info(f"Got {len(response)} logins for user {username} to be normalized")
+
+        except ConnectionError:
+            self.logger.error(f"Failed to establish a connection with host: {self.ingestion_config.get('host')}")
+        except TimeoutError:
+            self.logger.error(f"Timeout reached for the host: {self.ingestion_config.get('host')}")
+        except Exception as e:
+            self.logger.error(f"Exception while querying Splunk: {e}")
+        return response
diff --git a/buffalogs/impossible_travel/tests/test_splunk_ingestion.py b/buffalogs/impossible_travel/tests/test_splunk_ingestion.py
new file mode 100644
index 00000000..d155310c
--- /dev/null
+++ b/buffalogs/impossible_travel/tests/test_splunk_ingestion.py
@@ -0,0 +1,339 @@
+import json
+import os
+from datetime import datetime, timezone
+from unittest.mock import MagicMock, patch
+
+from django.conf import settings
+from django.test import TestCase
+from impossible_travel.ingestion.splunk_ingestion import SplunkIngestion
+
+
+def load_ingestion_config_data():
+    with open(
+        os.path.join(settings.CERTEGO_BUFFALOGS_CONFIG_PATH, "buffalogs/ingestion.json"),
+        mode="r",
+        encoding="utf-8",
+    ) as f:
+        config_ingestion = json.load(f)
+    return config_ingestion
+
+
+class SplunkIngestionTestCase(TestCase):
+    def setUp(self):
+        self.patcher = patch("splunklib.client.connect")
+        self.mock_connect = self.patcher.start()
+        self.mock_connect.return_value = MagicMock()
+        self.addCleanup(self.patcher.stop)
+
+        self.ingestion_config = load_ingestion_config_data()
+        self.splunk_config = self.ingestion_config["splunk"]
+        # Setup test data
+        self.user1_test_data = [
+            {
+                "user.name": "Stitch",
+                "@timestamp": "2025-02-26T13:40:15.173Z",
+                "_id": "log_id_0",
+                "index": "cloud-test_data",
+                "source.ip": "192.0.2.1",
+                "user_agent.original": "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0",
+                "source.as.organization.name": "",
+                "source.geo.location.lat": 37.2924,
+                "source.geo.location.lon": 136.2759,
+                "source.geo.country_name": "Japan",
+            },
+            {
+                "user.name": "Stitch",
+                "@timestamp": "2025-02-26T13:40:35.173Z",
+                "_id": "log_id_1",
+                "index": "cloud-test_data",
+                "source.ip": "192.0.2.1",
+                "user_agent.original": "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0",
+                "source.as.organization.name": "",
+                "source.geo.location.lat": 34.2924,
+                "source.geo.location.lon": 141.2759,
+                "source.geo.country_name": "Japan",
+            },
+        ]
+
+    @patch("splunklib.client.connect")
+    def test_init(self, mock_connect):
+        mock_service = MagicMock()
+        mock_connect.return_value = mock_service
+
+        ingestor = SplunkIngestion(self.splunk_config, mapping={})
+        mock_connect.assert_called_once_with(
+            host=self.splunk_config["host"],
+            port=self.splunk_config["port"],
+            username=self.splunk_config["username"],
+            password=self.splunk_config["password"],
+            scheme=self.splunk_config["scheme"],
+        )
+        self.assertEqual(ingestor.service, mock_service)
+
+    @patch("splunklib.client.connect")
+    def test_process_users(self, mock_connect):
+        mock_service = MagicMock()
+        mock_connect.return_value = mock_service
+        mock_job = MagicMock()
+        mock_job.is_done.return_value = True
+        mock_results = [
+            {"user.name": "Stitch"},
+            {"user.name": "scooby.doo@gmail.com"},
+        ]
+
+        mock_results_reader = MagicMock()
+        mock_results_reader.__iter__.return_value = mock_results
+
+        with patch("splunklib.results.ResultsReader", return_value=mock_results_reader):
+            mock_service.jobs.create.return_value = mock_job
+            ingestor = SplunkIngestion(self.splunk_config, mapping={})
+
+            start_date = datetime(2025, 2, 26, 13, 30, tzinfo=timezone.utc)
+            end_date = datetime(2025, 2, 26, 14, 0, tzinfo=timezone.utc)
+
+            result = ingestor.process_users(start_date, end_date)
+            self.assertCountEqual(result, ["Stitch", "scooby.doo@gmail.com"])
+            mock_service.jobs.create.assert_called_once()
+
+    @patch("splunklib.client.connect")
+    def test_process_users_connection_error(self, mock_connect):
+        mock_connect.side_effect = ConnectionError("Connection failed")
+        with self.assertLogs(level="ERROR") as cm:
+            ingestor = SplunkIngestion(self.splunk_config, mapping={})
+            result = ingestor.process_users(
+                start_date=datetime(2025, 2, 26, 13, 30, tzinfo=timezone.utc),
+                end_date=datetime(2025, 2, 26, 14, 0, tzinfo=timezone.utc),
+            )
+            self.assertEqual(result, [])
+        self.assertIn("Failed to establish a connection", cm.output[0])
+
+    @patch("splunklib.client.connect")
+    def test_process_user_logins(self, mock_connect):
+        mock_service = MagicMock()
+        mock_connect.return_value = mock_service
+        mock_job = MagicMock()
+        mock_job.is_done.return_value = True
+
+        mock_results_reader = MagicMock()
+        mock_results_reader.__iter__.return_value = self.user1_test_data
+
+        with patch("splunklib.results.ResultsReader", return_value=mock_results_reader):
+            mock_service.jobs.create.return_value = mock_job
+            ingestor = SplunkIngestion(self.splunk_config, mapping={})
+
+            start_date = datetime(2025, 2, 26, 13, 30, tzinfo=timezone.utc)
+            end_date = datetime(2025, 2, 26, 14, 0, tzinfo=timezone.utc)
+
+            result = ingestor.process_user_logins(start_date, end_date, "Stitch")
+            self.assertEqual(len(result), 2)
+            self.assertEqual(result[0]["user.name"], "Stitch")
+            mock_service.jobs.create.assert_called_once()
+
+    @patch("splunklib.client.connect")
+    def test_process_user_logins_empty_result(self, mock_connect):
+        mock_service = MagicMock()
+        mock_connect.return_value = mock_service
+        mock_job = MagicMock()
+        mock_job.is_done.return_value = True
+
+        mock_results_reader = MagicMock()
+        mock_results_reader.__iter__.return_value = []
+
+        with patch("splunklib.results.ResultsReader", return_value=mock_results_reader):
+            mock_service.jobs.create.return_value = mock_job
+            ingestor = SplunkIngestion(self.splunk_config, mapping={})
+
+            result = ingestor.process_user_logins(
+                start_date=datetime(2025, 2, 26, 13, 30, tzinfo=timezone.utc),
+                end_date=datetime(2025, 2, 26, 14, 0, tzinfo=timezone.utc),
+                username="Unknown",
+            )
+            self.assertEqual(result, [])
+
+    @patch("splunklib.client.connect")
+    def test_process_user_logins_missing_ip(self, mock_connect):
+        mock_service = MagicMock()
+        mock_connect.return_value = mock_service
+        mock_job = MagicMock()
+        mock_job.is_done.return_value = True
+
+        # Data missing source.ip (should be filtered out by Splunk query)
+        test_data = (
+            [
+                {
+                    "user.name": "Stitch",
+                    "@timestamp": "2025-02-26T13:40:15.173Z",
+                    "_id": "log_id_0",
+                    "index": "cloud-test_data",
+                    "user_agent.original": "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0",
+                    "source.as.organization.name": "",
+                    "source.geo.location.lat": 37.2924,
+                    "source.geo.location.lon": 136.2759,
+                    "source.geo.country_name": "Japan",
+                }
+            ],
+        )
+
+        mock_results_reader = MagicMock()
+        mock_results_reader.__iter__.return_value = test_data
+
+        with patch("splunklib.results.ResultsReader", return_value=mock_results_reader):
+            mock_service.jobs.create.return_value = mock_job
+            ingestor = SplunkIngestion(self.splunk_config, mapping={})
+            result = ingestor.process_user_logins(
+                start_date=datetime(2025, 2, 26, 13, 30, tzinfo=timezone.utc),
+                end_date=datetime(2025, 2, 26, 14, 0, tzinfo=timezone.utc),
+                username="Stitch",
+            )
+            self.assertEqual(result, [])
+
+    @patch("splunklib.client.connect")
+    def test_process_user_logins_field_mapping(self, mock_connect):
+        mock_service = MagicMock()
+        mock_connect.return_value = mock_service
+        mock_job = MagicMock()
+        mock_job.is_done.return_value = True
+
+        mock_results_reader = MagicMock()
+        mock_results_reader.__iter__.return_value = self.user1_test_data
+
+        with patch("splunklib.results.ResultsReader", return_value=mock_results_reader):
+            mock_service.jobs.create.return_value = mock_job
+            ingestor = SplunkIngestion(self.splunk_config, mapping={})
+            result = ingestor.process_user_logins(
+                start_date=datetime(2025, 2, 26, 13, 30, tzinfo=timezone.utc),
+                end_date=datetime(2025, 2, 26, 14, 0, tzinfo=timezone.utc),
+                username="Stitch",
+            )
+
+            self.assertEqual(len(result), len(self.user1_test_data))
+            for expected, actual in zip(self.user1_test_data, result):
+                for key in expected:
+                    self.assertIn(key, actual)
+                    self.assertEqual(actual[key], expected[key])
+
+    def test_normalize_fields_valid_data(self):
+        mapping = {
+            "user.name": "username",
+            "@timestamp": "timestamp",
+            "source.ip": "ip",
+            "source.geo.country_name": "country",
+            "source.geo.location.lat": "lat",
+            "source.geo.location.lon": "lon",
+            "user_agent.original": "user_agent",
+        }
+        ingestor = SplunkIngestion(self.splunk_config, mapping=mapping)
+        log_entry = {
+            "user": {"name": "test_user"},
+            "@timestamp": "2025-02-26T13:40:15.173Z",
+            "source": {
+                "ip": "192.168.1.1",
+                "geo": {
+                    "country_name": "Japan",
+                    "location": {
+                        "lat": 35.6762,
+                        "lon": 139.6503,
+                    },
+                },
+            },
+            "user_agent": {"original": "Mozilla/5.0"},
+        }
+        normalized = ingestor.normalize_fields([log_entry])
+        self.assertEqual(len(normalized), 1)
+        normalized_entry = normalized[0]
+        self.assertEqual(normalized_entry["username"], "test_user")
+        self.assertEqual(normalized_entry["timestamp"], "2025-02-26T13:40:15.173Z")
+        self.assertEqual(normalized_entry["ip"], "192.168.1.1")
+        self.assertEqual(normalized_entry["country"], "Japan")
+        self.assertEqual(normalized_entry["lat"], 35.6762)
+        self.assertEqual(normalized_entry["lon"], 139.6503)
+        self.assertEqual(normalized_entry["user_agent"], "Mozilla/5.0")
+
+    def test_normalize_fields_missing_optional(self):
+        """Test normalization when optional fields are missing"""
+        mapping = {
+            "user.name": "username",
+            "@timestamp": "timestamp",
+            "source.ip": "ip",
+            "source.geo.country_name": "country",
+            "source.geo.location.lat": "lat",
+            "source.geo.location.lon": "lon",
+            "user_agent.original": "user_agent",
+        }
+        ingestor = SplunkIngestion(self.splunk_config, mapping=mapping)
+        log_entry = {
+            "user": {"name": "test_user"},
+            "@timestamp": "2025-02-26T13:40:15.173Z",
+            "source": {
+                "ip": "192.168.1.1",
+                "geo": {
+                    "country_name": "Japan",
+                    "location": {
+                        "lat": 35.6762,
+                        "lon": 139.6503,
+                    },
+                },
+            },
+            # Missing user_agent.original
+        }
+        normalized = ingestor.normalize_fields([log_entry])
+        self.assertEqual(len(normalized), 1)
+        self.assertEqual(normalized[0]["user_agent"], "")
+
+    def test_normalize_fields_missing_required(self):
+        """Test normalization excludes entries with missing required fields"""
+        mapping = {
+            "user.name": "username",
+            "@timestamp": "timestamp",
+            "source.ip": "ip",
+            "source.geo.country_name": "country",
+            "source.geo.location.lat": "lat",
+            "source.geo.location.lon": "lon",
+        }
+        ingestor = SplunkIngestion(self.splunk_config, mapping=mapping)
+        log_entry = {
+            "user": {"name": "test_user"},
+            "@timestamp": "2025-02-26T13:40:15.173Z",
+            "source": {
+                # Missing ip
+                "geo": {
+                    "country_name": "Japan",
+                    "location": {
+                        "lat": 35.6762,
+                        "lon": 139.6503,
+                    },
+                },
+            },
+        }
+        normalized = ingestor.normalize_fields([log_entry])
+        self.assertEqual(len(normalized), 0)
+
+    def test_normalize_nested_fields(self):
+        """Test normalization correctly handles nested fields"""
+        mapping = {
+            "source.geo.location.lat": "lat",
+            "source.geo.location.lon": "lon",
+            "user.name": "username",
+            "@timestamp": "timestamp",
+            "source.ip": "ip",
+            "source.geo.country_name": "country",
+        }
+        ingestor = SplunkIngestion(self.splunk_config, mapping=mapping)
+        log_entry = {
+            "user": {"name": "test_user"},
+            "@timestamp": "2025-02-26T13:40:15.173Z",
+            "source": {
+                "ip": "192.168.1.1",
+                "geo": {
+                    "country_name": "Japan",
+                    "location": {
+                        "lat": 35.6762,
+                        "lon": 139.6503,
+                    },
+                },
+            },
+        }
+        normalized = ingestor.normalize_fields([log_entry])
+        self.assertEqual(len(normalized), 1)
+        self.assertEqual(normalized[0]["lat"], 35.6762)
+        self.assertEqual(normalized[0]["lon"], 139.6503)
diff --git a/buffalogs/requirements.txt b/buffalogs/requirements.txt
index 37f50a41..cf93e0c9 100644
--- a/buffalogs/requirements.txt
+++ b/buffalogs/requirements.txt
@@ -31,3 +31,4 @@ uWSGI>=2.0.28
 virtualenv>=20.17.1
 wcwidth>=0.2.5
 requests>=2.32.3    # for Alerting module
+splunk-sdk>=2.1.0   # for Splunk ingestion source (optional)
\ No newline at end of file
diff --git a/config/buffalogs/ingestion.json b/config/buffalogs/ingestion.json
index 234d0bf2..f12d7431 100644
--- a/config/buffalogs/ingestion.json
+++ b/config/buffalogs/ingestion.json
@@ -41,9 +41,25 @@
         }
     },
     "splunk": {
-        "url": "http://localhost:9200/",
+        "host": "splunk",
+        "port": 8089,
+        "scheme": "http",
         "username": "foobar",
         "password": "bar",
-        "custom_mapping": {}
-    }
+        "timeout": 90,
+        "indexes": "main",
+        "bucket_size": 10000,
+        "custom_mapping": {
+            "user.name": "username",
+            "@timestamp": "timestamp",
+            "source.ip": "ip",
+            "source.geo.country_name": "country",
+            "source.geo.location.lat": "lat",
+            "source.geo.location.lon": "lon",
+            "user_agent.original": "agent",
+            "source.as.organization.name": "organization",
+            "_id": "id",
+            "index": "index"
+        }
+      }
 }
\ No newline at end of file
diff --git a/docker-compose.splunk.yaml b/docker-compose.splunk.yaml
new file mode 100644
index 00000000..65b1d9dc
--- /dev/null
+++ b/docker-compose.splunk.yaml
@@ -0,0 +1,33 @@
+version: '3.8'
+
+services:
+    splunk:
+        container_name: buffalogs_splunk
+        image: splunk/splunk:latest
+        hostname: splunk
+        environment:
+            - SPLUNK_START_ARGS=--accept-license
+            - SPLUNK_PASSWORD=YourSecurePassword
+            - SPLUNK_ENABLE_LISTEN=9997
+            - SPLUNK_ADD=tcp 1514
+        ports:
+            - "8000:8000"   # Splunk Web UI
+            - "8089:8089"   # Splunk REST API
+            - "9997:9997"   # Splunk indexer receiving port
+            - "1514:1514"   # TCP input for log forwarding
+        volumes:
+            - splunk_data:/opt/splunk/var
+            - splunk_etc:/opt/splunk/etc
+        networks:
+            - splunk-net
+        restart: unless-stopped
+
+volumes:
+    splunk_data:
+        driver: local
+    splunk_etc:
+        driver: local
+
+networks:
+    splunk-net:
+        driver: bridge
diff --git a/docs/ingestion/splunk.md b/docs/ingestion/splunk.md
new file mode 100644
index 00000000..c908f1a0
--- /dev/null
+++ b/docs/ingestion/splunk.md
@@ -0,0 +1,189 @@
+# Splunk Ingestion Documentation
+
+This document outlines a step-by-step guide for configuring Splunk using Docker services. It walks you through deploying Splunk in a containerized environment, then explains how you can send data in two ways: locally (by manually adding data via the Splunk web interface) and programmatically (via API calls, like using the HTTP Event Collector). Finally, it describes how Splunk processes the incoming data, parsing and indexing it according to defined configurations.
+
+---
+
+## Pre-requisites
+
+- **Docker & Docker Compose:** Make sure you have Docker and Docker Compose installed.
+- **Basic Splunk Knowledge:** Familiarity with Splunk’s interface and basic ingestion concepts.
+
+---
+
+## Customization of docker-compose.splunk.yaml
+
+In the `docker-compose.splunk.yaml` file, several configurations can be tailored to meet your specific needs:
+
+- **Default Password:**  
+  Change the default password to enhance security. Look for the environment variable (e.g., `SPLUNK_PASSWORD`) and update its value.
+  
+- **Ports and Volumes:**  
+  Customize port mappings and volume mounts if your environment requires different configurations.
+  
+> **Tip:** Open the YAML file and look for sections marked with comments like `# Customize below` to find where you might change passwords or ports.
+
+---
+
+## Running the Docker Compose File
+
+To run your Splunk service:
+
+1. **Navigate to the Project Directory:**  
+   Open your terminal and navigate to the directory containing your `docker-compose.splunk.yaml` file.
+
+2. **Run Docker Compose:**  
+   Execute the following command:  
+   ```
+   docker-compose -f docker-compose.splunk.yaml up -d
+   ```  
+   This command will start the Splunk service in detached mode.
+
+3. **Verify Splunk is Running:**  
+   Open your browser and navigate to [http://localhost:8000](http://localhost:8000) (or your custom port if modified) to access the Splunk web interface.
+
+---
+
+## Configuring Ingestion Settings
+
+The ingestion configuration is maintained in a separate JSON file (e.g., `ingestion.json`). Here, you can define how data should be processed. At this stage, the installation is basic, but you can update the file as your needs.
+
+- **Custom Fields:**  
+  Modify the JSON keys and values to reflect the fields you need. This is where you align with your internal schema requirements.
+
+> **Example Placeholder:**  
+> Inside your `ingestion.json`, you can configure parameters like index,username etc.
+
+---
+
+## Logs Structure and Data Flow
+
+Splunk accepts data that follows a defined structure. For instance, our system expects logs to comply with the Elastic Common Schema. Below is an example schema:
+
+```json
+{
+    "index": "<elastic_index>",
+    "id": "<log_id>",
+    "timestamp": "<log_timestamp>",
+    "user": {
+        "name": "<user_name>"
+    },
+    "source": {
+        "geo": {
+            "country_name": "<country_origin_log>",
+            "location": {
+                "lat": "<log_latitude>",
+                "lon": "<log_longitude>"
+            }
+        },
+        "as": {
+            "organization": {
+                "name": "<ISP_name>"
+            }
+        },
+        "ip": "<log_source_ip>",
+        "intelligence_category": "<intelligence_category>"
+    },
+    "user_agent": {
+        "original": "<log_device_user_agent>"
+    },
+    "event": {
+        "type": "start",
+        "category": "authentication",
+        "outcome": "success"
+    }
+}
+```
+---
+
+### **Adding Data to Local Splunk**  
+
+If you're running Splunk locally, you can add data using the **GUI (manual upload)** or **HEC (automated ingestion via API calls).**  
+
+### **Method 1: Manual Upload via Splunk GUI**  
+1. **Open Splunk Web Interface**  
+   - Go to [http://localhost:8000](http://localhost:8000) in your browser.  
+   - Log in with your credentials.
+
+2. **Navigate to the Data Input Section**  
+   - Click on **"Settings" > "Add Data"**  
+   - Choose **Upload files from my computer** (if you have a local log file).  
+
+3. **Upload Your File**  
+   - Click **Browse**, select your file, and upload it.  
+   - Choose a **Source Type** (or let Splunk auto-detect).  
+   - Assign an **Index** (default is `main`).  
+
+4. **Finish Setup & Start Searching**  
+   - Click **Next**, review settings, and click **Submit**.  
+   - Go to **Search & Reporting** to query your data.  
+
+---
+
+### **Method 2: Send Data Using HEC (HTTP Event Collector)**
+To send data programmatically, ensure HEC is enabled on your local Splunk:
+
+#### **1. Enable HEC in Splunk**
+- Go to **Settings > Data Inputs > HTTP Event Collector**.
+- Click **New Token**, configure the **source type**, and **enable it**.
+- Note the generated **HEC Token**.
+
+#### **2. Send Data via cURL**
+Run the following command in your terminal:
+
+```bash
+curl -k https://localhost:8088/services/collector \
+     -H "Authorization: Splunk <your_hec_token>" \
+     -H "Content-Type: application/json" \
+     -d '{"event": "Hello, Splunk!", "sourcetype": "manual"}'
+```
+
+**Replace `<your_hec_token>` with your actual token.**  
+
+---
+
+
+**How It Works:**
+
+> **Note:** In this documentation, we are using a cloud-based example, but the process and concepts are the same for a local Splunk installation. The only difference is that you will need to replace the `<splunk_host>` with `localhost` if running Splunk locally.
+
+> 
+> ![Raw data](https://github.com/user-attachments/assets/e0863604-4480-4468-a7a8-26222b199106)
+
+
+- **Data Pasting:**  
+  When you paste data with this schema, Splunk intelligently parses the nested structure and automatically generates table-based values.
+  
+> **Table view of Data:**  
+> ![Screenshot of ingestion source table](https://github.com/user-attachments/assets/821ee11f-7e10-44d9-9baf-fd01a32826c5)
+
+
+ **Processing Logic:**
+  
+  The `splunk_ingestion.py` connector is built around two core functions:
+  
+  - **process_user:** Extracts and handles the username data.
+  - **process_user_login:** Collects and processes login responses.
+
+This streamlined approach not only organizes your log ingestion flow but also sets the stage for additional processing.
+
+---
+
+## Troubleshooting and Support
+
+If you encounter any issues:
+
+- **Raise an Issue/Discussion:**  
+  Please report your problems or discuss concerns on the [BuffaLogs Official Repository](https://github.com/buffalogs/buffalogs). This ensures you receive timely support from the community.
+  
+- **Official Splunk Documentation:**  
+  For more detailed troubleshooting regarding Splunk’s ingestion processes, check out the [Splunk Ingestion Documentation](https://docs.splunk.com/Documentation/Splunk).
+
+---
+
+Remember, this guide is a starting point for setting up a basic Docker-based Splunk installation. As your needs evolve, you can further customize both the `docker-compose.splunk.yaml` and `ingestion.json` configurations.
+
+If you feel stuck at any point, don't hesitate to seek help on the official repository or engage in community discussions. 
+
+---
+


Reviews:
user: login: Lorygold
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T08:28:45Z
user: login: Lorygold
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T08:31:00Z
user: login: Lorygold
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T08:33:05Z
user: login: drona-gyawali
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T11:55:07Z
user: login: drona-gyawali
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T11:56:44Z
user: login: drona-gyawali
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T11:58:17Z
user: login: Lorygold
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T15:31:05Z
user: login: ManofWax
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T10:39:40Z

Review comments:
diff_hunk: @@ -43,5 +44,7 @@ def get_ingestion_class(self):
                 return ElasticsearchIngestion(self.ingestion_config, self.mapping)
             case BaseIngestion.SupportedIngestionSources.OPENSEARCH:
                 return OpensearchIngestion(self.ingestion_config)
+            case BaseIngestion.SupportedIngestionSources.SPLUNK:
+                return SplunkIngestion(self.ingestion_config)
path: buffalogs/impossible_travel/ingestion/ingestion_factory.py
user: login: Lorygold
body: you should pass also the mapping for the instance creation, right?
created_at: 2025-04-08T08:28:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,223 @@
+import json
+import os
+from datetime import datetime, timezone
+from unittest.mock import MagicMock, patch
+
+from django.conf import settings
+from django.test import TestCase
+from impossible_travel.ingestion.splunk_ingestion import SplunkIngestion
+
+
+def load_ingestion_config_data():
+    with open(
+        os.path.join(settings.CERTEGO_BUFFALOGS_CONFIG_PATH, "buffalogs/ingestion.json"),
+        mode="r",
+        encoding="utf-8",
+    ) as f:
+        config_ingestion = json.load(f)
+    return config_ingestion
+
+
+def load_test_data(name):
+    with open(os.path.join(settings.CERTEGO_DJANGO_PROJ_BASE_DIR, "impossible_travel/tests/test_data/", name + ".json")) as file:
+        data = json.load(file)
+    return data
+
+
+class SplunkIngestionTestCase(TestCase):
+    def setUp(self):
+        self.ingestion_config = load_ingestion_config_data()
+        self.splunk_config = {
path: buffalogs/impossible_travel/tests/test_splunk_ingestion.py
user: login: Lorygold
body: you could take the `splunk_config` from the `ingestion.json` file, in order to test if also the configuration file is correct
created_at: 2025-04-08T08:31:00Z
author_association: CONTRIBUTOR
path: buffalogs/impossible_travel/tests/test_splunk_ingestion.py
user: login: Lorygold
body: could you also add some tests for the `normalize_fields` function? Even if it's concrete, for checking if it works also with Splunk
created_at: 2025-04-08T08:33:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -43,5 +44,7 @@ def get_ingestion_class(self):
                 return ElasticsearchIngestion(self.ingestion_config, self.mapping)
             case BaseIngestion.SupportedIngestionSources.OPENSEARCH:
                 return OpensearchIngestion(self.ingestion_config)
+            case BaseIngestion.SupportedIngestionSources.SPLUNK:
+                return SplunkIngestion(self.ingestion_config)
path: buffalogs/impossible_travel/ingestion/ingestion_factory.py
user: login: drona-gyawali
body: Yes, I have added the mapping. 
created_at: 2025-04-08T11:55:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,223 @@
+import json
+import os
+from datetime import datetime, timezone
+from unittest.mock import MagicMock, patch
+
+from django.conf import settings
+from django.test import TestCase
+from impossible_travel.ingestion.splunk_ingestion import SplunkIngestion
+
+
+def load_ingestion_config_data():
+    with open(
+        os.path.join(settings.CERTEGO_BUFFALOGS_CONFIG_PATH, "buffalogs/ingestion.json"),
+        mode="r",
+        encoding="utf-8",
+    ) as f:
+        config_ingestion = json.load(f)
+    return config_ingestion
+
+
+def load_test_data(name):
+    with open(os.path.join(settings.CERTEGO_DJANGO_PROJ_BASE_DIR, "impossible_travel/tests/test_data/", name + ".json")) as file:
+        data = json.load(file)
+    return data
+
+
+class SplunkIngestionTestCase(TestCase):
+    def setUp(self):
+        self.ingestion_config = load_ingestion_config_data()
+        self.splunk_config = {
path: buffalogs/impossible_travel/tests/test_splunk_ingestion.py
user: login: drona-gyawali
body: Yes, I did. 
created_at: 2025-04-08T11:56:44Z
author_association: CONTRIBUTOR
path: buffalogs/impossible_travel/tests/test_splunk_ingestion.py
user: login: drona-gyawali
body: Yes, I added some tescases for the normalize_fields.
created_at: 2025-04-08T11:58:16Z
author_association: CONTRIBUTOR

PR comments:
user: login: Lorygold
created_at: 2025-04-07T16:36:18Z
body: Hi @drona-gyawali , a refactoring was needed on the ignestion flow. In particular, as you can see in the #233 merged PR, the init classes have changed and the `normalize_fields` function become concrete (it's not abstract anymore).
So, probably also your code need a refactor, too. Sorry for that, but it was required because it's more correct with the abstract concept


author_association: CONTRIBUTOR
user: login: drona-gyawali
created_at: 2025-04-07T16:45:28Z
body: Thank you for pointing that out. I’ll make sure to refactor the code shortly to improve its structure and maintainability.
author_association: CONTRIBUTOR
user: login: drona-gyawali
created_at: 2025-04-08T02:59:14Z
body: The requested changes have been implemented. Please let me know if anything else is required. The PR is ready for your review.

author_association: CONTRIBUTOR
user: login: Lorygold
created_at: 2025-04-08T15:34:10Z
body: I think now your code is well done, let's wait for @ManofWax review, too
author_association: CONTRIBUTOR