Repository: satomic/copilot-usage-advanced-dashboard
Pull request #19: Add Elasticsearch authentication
Author: login: rprouse
Created: 2025-04-10T14:25:40Z | Merged: 2025-04-10T16:00:54Z

I am using an authenticated version of Elasticsearch. This adds the ability to pass in `ELASTICSEARCH_USER` and `ELASTICSEARCH_PASS`. If they are not set, it falls back on the original non-authenticated Elasticsearch.

If you want to test, I have an image on DockerHub, `rprouse/cpuad-updater:20250410`



Diff:
diff --git a/.gitignore b/.gitignore
index dcc693a..6d5e7f7 100644
--- a/.gitignore
+++ b/.gitignore
@@ -24,4 +24,7 @@ dashboard-model-*
 .vscode/
 *bak*
 
-debug_*
\ No newline at end of file
+debug_*
+
+# .env files
+*.env
diff --git a/README.md b/README.md
index fb303c6..8b7bde6 100644
--- a/README.md
+++ b/README.md
@@ -565,8 +565,8 @@ drwxr-xr-x 2 root root  4096 Dec 17 00:18 grafana
 
 Parameter description
 
-- `GITHUB_PAT`: 
-  - Your GitHub account needs to have Owner permissions for Organizations. 
+- `GITHUB_PAT`:
+  - Your GitHub account needs to have Owner permissions for Organizations.
   - [Create a personal access token (classic)](https://docs.github.com/en/enterprise-cloud@latest/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token-classic) of your account with the `manage_billing:copilot`, `read:enterprise`, `read:org` scope. Then please replace `<YOUR_GITHUB_PAT>` with the actual PAT.
   - If you encounter PAT permission error, please **Allow access via fine-grained personal access tokens** in Organization's **Settings** - **Personal access tokens**.
 - `ORGANIZATION_SLUGS`: The Slugs of all Organizations that you want to monitor, which can be one or multiple separated by `,` (English symbol). **If you are using Copilot Standalone, use your Standalone Slug here, prefixed with `standalone:`, for example `standalone:YOUR_STANDALONE_SLUG`**. Please replace `<YOUR_ORGANIZATION_SLUGS>` with the actual value. For example, the following types of values are supported:
@@ -591,6 +591,13 @@ docker run -itd \
 satomic/cpuad-updater
 ```
 
+If your Elasticsearch instance requires authentication, pass include the `ELASTICSEARCH_USER` and `ELASTICSEARCH_PASS` environment variables.
+
+```bash
+-e ELASTICSEARCH_USER="elastic"
+-e ELASTICSEARCH_PASS="mypassword"
+```
+
 ### Option 2. Run in source code mode
 
 1. Confirm that you are in the correct path
diff --git a/main.py b/main.py
index b2445c7..dcbbf77 100644
--- a/main.py
+++ b/main.py
@@ -33,7 +33,9 @@ def date_str():
     # ElasticSearch
     primary_key = os.getenv('PRIMARY_KEY', 'unique_hash')
     elasticsearch_url = os.getenv('ELASTICSEARCH_URL', 'http://localhost:9200')
-    
+    elasticsearch_user = os.getenv('ELASTICSEARCH_USER', None)
+    elasticsearch_pass = os.getenv('ELASTICSEARCH_PASS', None)
+
     # Log path
     log_path = os.getenv('LOG_PATH', 'logs')
 
@@ -154,7 +156,7 @@ def __init__(self, token, enterprise_slug, save_to_json=True):
         logger.info(f"Initialized GitHubEnterpriseManager for enterprise: {enterprise_slug}")
 
     def _fetch_all_organizations(self, save_to_json=False):
-        
+
         # GraphQL query
         query = '''
         {
@@ -256,7 +258,7 @@ def get_seat_info_settings_standalone(self, save_to_json=True):
         data_seats = github_api_request_handler(url, error_return_value={})
         if not data_seats:
             return data_seats
-        
+
         data = {
             "seat_management_setting": "assign_selected",
             "public_code_suggestions": "allow",
@@ -279,11 +281,11 @@ def get_seat_info_settings_standalone(self, save_to_json=True):
                 start_of_yesterday = (datetime.now(created_date.tzinfo).replace(hour=0, minute=0, second=0, microsecond=0) - timedelta(days=1))
                 if created_date >= start_of_yesterday:
                     data['seat_added_this_cycle'] += 1
-            
+
             seat_pending_cancellation_date = data_seat.get('pending_cancellation_date')
             if seat_pending_cancellation_date:
                 data['seat_pending_cancellation'] += 1
-            
+
             seat_last_activity_at = data_seat.get('last_activity_at')
             if seat_last_activity_at:
                 last_activity_date = datetime.strptime(seat_last_activity_at, '%Y-%m-%dT%H:%M:%S%z')
@@ -297,7 +299,7 @@ def get_seat_info_settings_standalone(self, save_to_json=True):
         data['organization_slug'] = self.organization_slug
         data['day'] = current_time()[:10]
         data['unique_hash'] = generate_unique_hash(
-            data, 
+            data,
             key_properties=['organization_slug', 'day']
         )
 
@@ -354,7 +356,7 @@ def get_seat_info_settings(self, save_to_json=True):
         data['organization_slug'] = self.organization_slug
         data['day'] = current_time()[:10]
         data['unique_hash'] = generate_unique_hash(
-            data, 
+            data,
             key_properties=['organization_slug', 'day']
         )
 
@@ -388,15 +390,15 @@ def get_seat_assignments(self, save_to_json=True):
                 seat['assignee_team_slug'] = seat.get('assigning_team', {}).get('slug', 'no-team')
                 seat['assignee_team_html_url'] = seat.get('assigning_team', {}).get('html_url')
                 seat.pop('assigning_team', None)
-                
+
                 seat['organization_slug'] = self.organization_slug
                 # seat['day'] = current_time()[:10] # 2025-04-02T08:00:00+08:00 seat['updated_at'][:10]
                 seat['day'] = datetime.now(datetime.strptime(seat['updated_at'], '%Y-%m-%dT%H:%M:%S%z').tzinfo).strftime('%Y-%m-%d %H:%M:%S.%f')[:10]
                 seat['unique_hash'] = generate_unique_hash(
-                    seat, 
+                    seat,
                     key_properties=['organization_slug', 'assignee_login', 'day']
                 )
-                
+
                 last_activity_at = seat.get('last_activity_at') # 2025-04-02T00:22:35+08:00
                 if last_activity_at:
                     last_activity_date = datetime.strptime(last_activity_at, '%Y-%m-%dT%H:%M:%S%z')
@@ -418,7 +420,7 @@ def get_seat_assignments(self, save_to_json=True):
 
     def _fetch_all_teams(self, save_to_json=True):
         # Teams under the same org are essentially at the same level because the URL does not reflect the nested relationship, so team names cannot be duplicated
-        
+
         url = f"https://api.github.com/{self.api_type}/{self.organization_slug}/teams"
         teams = []
         page = 1
@@ -436,7 +438,7 @@ def _fetch_all_teams(self, save_to_json=True):
                 break
             teams.extend(page_teams)
             page += 1  # 获取下一页数据
-        
+
         teams = self._add_fullpath_slug(teams)
         teams = assign_position_in_tree(teams)
         dict_save_to_json_file(teams, f'{self.organization_slug}_all_teams', save_to_json=save_to_json)
@@ -478,7 +480,7 @@ def get_total_list(self):
             total_data.pop('breakdown_chat', None)
             total_data = total_data | self.additional_properties
             total_data['unique_hash'] = generate_unique_hash(
-                total_data, 
+                total_data,
                 key_properties=['organization_slug', 'team_slug', 'day']
             )
 
@@ -509,7 +511,7 @@ def get_breakdown_list(self):
                 #     breakdown_entry_with_day['language'] = 'json'
 
                 breakdown_entry_with_day['unique_hash'] = generate_unique_hash(
-                    breakdown_entry_with_day, 
+                    breakdown_entry_with_day,
                     key_properties=['organization_slug', 'team_slug', 'day', 'language', 'editor', 'model']
                 )
 
@@ -547,9 +549,15 @@ class ElasticsearchManager:
 
     def __init__(self, primary_key=Paras.primary_key):
         self.primary_key = primary_key
-        self.es = Elasticsearch(
-            Paras.elasticsearch_url
-        )
+        if Paras.elasticsearch_user is None or Paras.elasticsearch_pass is None:
+            logger.info(f"Using Elasticsearch without authentication")
+            self.es = Elasticsearch(Paras.elasticsearch_url)
+        else:
+            logger.info(f"Using basic authentication for Elasticsearch")
+            self.es = Elasticsearch(
+                Paras.elasticsearch_url,
+                basic_auth=(Paras.elasticsearch_user, Paras.elasticsearch_pass)
+            )
         self.check_and_create_indexes()
 
     # Check if all indexes in the indexes are present, and if they don't, they are created based on the files in the mapping folder
@@ -574,7 +582,7 @@ def write_to_es(self, index_name, data, update_condition=None):
         try:
             # Get existing document
             existing_doc = self.es.get(index=index_name, id=doc_id)
-            
+
             # Check update condition if provided
             if update_condition:
                 should_preserve_fields = True
@@ -582,20 +590,20 @@ def write_to_es(self, index_name, data, update_condition=None):
                     if field not in existing_doc['_source'] or existing_doc['_source'][field] != value:
                         should_preserve_fields = False
                         break
-                
+
                 if should_preserve_fields:
                     # Preserve fields listed in update_condition by copying their values from existing document
                     for field in update_condition.keys():
                         if field in existing_doc['_source']:
                             data[field] = existing_doc['_source'][field]
                     logger.info(f'[partial update] to [{index_name}]: {doc_id} - preserving fields: {list(update_condition.keys())}')
-            
+
             # Always update document, possibly with some preserved fields
             self.es.update(index=index_name, id=doc_id, doc=data)
             logger.info(f'[updated] to [{index_name}]: {data}')
         except NotFoundError:
             self.es.index(index=index_name, id=doc_id, document=data)
-            logger.info(f'[created] to [{index_name}]: {data}') 
+            logger.info(f'[created] to [{index_name}]: {data}')
 
 def main(organization_slug):
     logger.info(f"==========================================================================================================")
@@ -609,7 +617,7 @@ def main(organization_slug):
     organization_slug = organization_slug.replace('standalone:', '')
 
     logger.info(f"Starting data processing for {slug_type}: {organization_slug}")
-    github_org_manager = GitHubOrganizationManager(organization_slug, is_standalone=is_standalone) 
+    github_org_manager = GitHubOrganizationManager(organization_slug, is_standalone=is_standalone)
     es_manager = ElasticsearchManager()
 
 
@@ -659,7 +667,7 @@ def main(organization_slug):
         # and save to json file
         total_list = data_splitter.get_total_list()
         dict_save_to_json_file(total_list, f'{team_slug}_total_list')
-        
+
         breakdown_list = data_splitter.get_breakdown_list()
         dict_save_to_json_file(breakdown_list, f'{team_slug}_breakdown_list')
 
@@ -669,19 +677,19 @@ def main(organization_slug):
         # Write to ES
         for total_data in total_list:
             es_manager.write_to_es(Indexes.index_name_total, total_data)
-        
+
         for breakdown_data in breakdown_list:
             es_manager.write_to_es(Indexes.index_name_breakdown, breakdown_data)
 
         for breakdown_chat_data in breakdown_chat_list:
             es_manager.write_to_es(Indexes.index_name_breakdown_chat, breakdown_chat_data)
-        
+
         logger.info(f"Data processing completed for team: {team_slug}")
 
 
 
 if __name__ == '__main__':
-    
+
     while True:
         try:
             # Split Paras.organization_slugs and process each organization, remember to remove spaces after splitting


Reviews:
user: login: copilot-pull-request-reviewer[bot]
body: 

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.




state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T15:57:36Z

Review comments:
diff_hunk: @@ -547,9 +549,15 @@ class ElasticsearchManager:
 
     def __init__(self, primary_key=Paras.primary_key):
         self.primary_key = primary_key
-        self.es = Elasticsearch(
-            Paras.elasticsearch_url
-        )
+        if Paras.elasticsearch_user is None or Paras.elasticsearch_pass is None:
path: main.py
user: login: Copilot
body: The current check treats the scenario where only one credential is provided as if authentication is not needed, which could lead to unexpected behavior. Consider adding a validation that requires both ELASTICSEARCH_USER and ELASTICSEARCH_PASS to be set, or raise an error if only one is provided.
```suggestion
        if (Paras.elasticsearch_user is None) != (Paras.elasticsearch_pass is None):
            logger.error("Both ELASTICSEARCH_USER and ELASTICSEARCH_PASS must be set, or neither should be set.")
            raise ValueError("Invalid Elasticsearch credentials configuration.")
        if Paras.elasticsearch_user is None and Paras.elasticsearch_pass is None:
```
created_at: 2025-04-10T15:57:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -591,6 +591,13 @@ docker run -itd \
 satomic/cpuad-updater
 ```
 
+If your Elasticsearch instance requires authentication, pass include the `ELASTICSEARCH_USER` and `ELASTICSEARCH_PASS` environment variables.
path: README.md
user: login: Copilot
body: [nitpick] Consider rephrasing the sentence for clarity, e.g., 'If your Elasticsearch instance requires authentication, include the `ELASTICSEARCH_USER` and `ELASTICSEARCH_PASS` environment variables.'
```suggestion
If your Elasticsearch instance requires authentication, include the `ELASTICSEARCH_USER` and `ELASTICSEARCH_PASS` environment variables.
```
created_at: 2025-04-10T15:57:36Z
author_association: CONTRIBUTOR

PR comments:
user: login: satomic
created_at: 2025-04-10T16:00:49Z
body: reviewed, merged
author_association: OWNER
user: login: rprouse
created_at: 2025-04-10T19:29:01Z
body: I pulled your latest docker image. Both of my issues are fixed. Thanks for your awesome work.
author_association: CONTRIBUTOR