Repository: celery/django-celery-beat
Pull request #900: refactor: use in operator instead of regex operator in crontab query to fix mssql regression
Author: login: alirafiei75
Created: 2025-05-16T18:11:20Z | Merged: 2025-05-18T03:19:59Z

## Description

This pull request refactors _get_crontab_exclude_query method of DatabaseScheduler class to use a list of valid numeric hours for crontab query instead of regex pattern

## Related Issues

Fixes: #897  


Diff:
diff --git a/django_celery_beat/schedulers.py b/django_celery_beat/schedulers.py
index dce0a077..99d98f8c 100644
--- a/django_celery_beat/schedulers.py
+++ b/django_celery_beat/schedulers.py
@@ -314,13 +314,10 @@ def _get_crontab_exclude_query(self):
         ]
         hours_to_include += [4]  # celery's default cleanup task
 
-        # Regex pattern to match only numbers
-        # This ensures we only process numeric hour values
-        numeric_hour_pattern = r'^\d+$'
-
         # Get all tasks with a simple numeric hour value
+        valid_numeric_hours = self._get_valid_hour_formats()
         numeric_hour_tasks = CrontabSchedule.objects.filter(
-            hour__regex=numeric_hour_pattern
+            hour__in=valid_numeric_hours
         )
 
         # Annotate these tasks with their server-hour equivalent
@@ -359,6 +356,15 @@ def _get_crontab_exclude_query(self):
 
         return exclude_query
 
+    def _get_valid_hour_formats(self):
+        """
+        Return a list of all valid hour values (0-23).
+        Both zero-padded ("00"–"09") and non-padded ("0"–"23")
+        """
+        return [str(hour) for hour in range(24)] + [
+            f"{hour:02d}" for hour in range(10)
+        ]
+
     def _get_unique_timezone_names(self):
         """Get a list of all unique timezone names used in CrontabSchedule"""
         return CrontabSchedule.objects.values_list(
diff --git a/t/unit/test_schedulers.py b/t/unit/test_schedulers.py
index 6ccc88bf..3a9d757d 100644
--- a/t/unit/test_schedulers.py
+++ b/t/unit/test_schedulers.py
@@ -1212,6 +1212,41 @@ def test_crontab_timezone_conversion_with_negative_offset_and_dst(
             "Paris outside window task should be excluded"
         )
 
+    def test_scheduler_valid_hours(self):
+        """Test the _get_valid_hour_formats method."""
+        # Create an instance of DatabaseScheduler
+        s = self.Scheduler(app=self.app)
+
+        # Get the valid hours list
+        valid_hours = s._get_valid_hour_formats()
+
+        # Basic validations
+        # 24 regular hours + 10 zero-padded hours
+        assert len(valid_hours) == 34
+
+        # Check both regular and zero-padded formats are present
+        assert "0" in valid_hours
+        assert "00" in valid_hours
+        assert "9" in valid_hours
+        assert "09" in valid_hours
+        assert "23" in valid_hours
+
+        # Verify all hours 0-23 are included
+        for hour in range(24):
+            assert str(hour) in valid_hours
+
+        # Verify zero-padded hours 00-09 are included
+        for hour in range(10):
+            assert f"{hour:02d}" in valid_hours
+
+        # Check for duplicates (set should have same length as list)
+        assert len(set(valid_hours)) == len(valid_hours)
+
+        # Verify all entries are valid hour representations
+        for hour_str in valid_hours:
+            hour_value = int(hour_str)
+            assert 0 <= hour_value <= 23
+
 
 @pytest.mark.django_db
 class test_models(SchedulerCase):


Reviews:
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This PR refactors the `_get_crontab_exclude_query` method in `DatabaseScheduler` to replace a regex-based hour filter with an explicit `__in` lookup over a generated list of valid hour strings.

- Removed the `numeric_hour_pattern` regex filter.
- Introduced a `valid_hours` list combining non-padded and zero-padded representations for hours 0–23.
- Updated the queryset to use `hour__in=valid_hours` instead of `hour__regex`.


<details>
<summary>Comments suppressed due to low confidence (1)</summary>

**django_celery_beat/schedulers.py:323**
* Add unit tests to verify that the filter correctly matches both padded and non-padded hour values in the database, ensuring no valid entries are missed.
```
hour__in=valid_hours
```
</details>


state: COMMENTED
author_association: NONE
submitted_at: 2025-05-17T05:01:54Z
user: login: auvipy
body: can we verify / fix this with modified tests please? the copilot suggestion seems to break the tests
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-05-17T08:27:01Z
user: login: auvipy
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-18T03:17:44Z

Review comments:
diff_hunk: @@ -314,13 +314,13 @@ def _get_crontab_exclude_query(self):
         ]
         hours_to_include += [4]  # celery's default cleanup task
 
-        # Regex pattern to match only numbers
-        # This ensures we only process numeric hour values
-        numeric_hour_pattern = r'^\d+$'
-
         # Get all tasks with a simple numeric hour value
+        # Create a list of all valid hour values (0-23)
+        valid_hours = [str(hour) for hour in range(24)] + [
+            f"{hour:02d}" for hour in range(10)
+        ]
path: django_celery_beat/schedulers.py
user: login: Copilot
body: [nitpick] Consider simplifying the generation of both padded and non-padded hour strings using a single comprehension, e.g., `[str(hour) for hour in range(24)] + [str(hour).zfill(2) for hour in range(24)]`, to reduce mental overhead for future maintainers.
```suggestion
        # Create a list of all valid hour values (0-23), both padded and non-padded
        valid_hours = [str(hour) if hour >= 10 else f"{hour:02d}" for hour in range(24)]
```
created_at: 2025-05-17T05:01:54Z
author_association: NONE
diff_hunk: @@ -314,13 +314,13 @@ def _get_crontab_exclude_query(self):
         ]
         hours_to_include += [4]  # celery's default cleanup task
 
-        # Regex pattern to match only numbers
-        # This ensures we only process numeric hour values
-        numeric_hour_pattern = r'^\d+$'
-
         # Get all tasks with a simple numeric hour value
+        # Create a list of all valid hour values (0-23)
path: django_celery_beat/schedulers.py
user: login: Copilot
body: [nitpick] Expand this comment to clarify why both zero-padded ("00"–"09") and non-padded ("0"–"23") formats are needed, to help future readers understand database storage variations.
```suggestion
        # Create a list of all valid hour values (0-23).
        # Both zero-padded ("00"–"09") and non-padded ("0"–"23") formats are included
        # to account for variations in how hour values are stored in the database.
        # Some databases or configurations may store single-digit hours without padding,
        # while others may use zero-padded strings. Including both ensures compatibility.
```
created_at: 2025-05-17T05:01:54Z
author_association: NONE

PR comments:
user: login: codecov[bot]
created_at: 2025-05-16T18:15:02Z
body: ## [Codecov](https://app.codecov.io/gh/celery/django-celery-beat/pull/900?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=celery) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 87.67%. Comparing base [(`673dbc5`)](https://app.codecov.io/gh/celery/django-celery-beat/commit/673dbc5590288f5a1f610c4569d4f4c797abefe0?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=celery) to head [(`e05c85a`)](https://app.codecov.io/gh/celery/django-celery-beat/commit/e05c85ae2267ce33ff4efe2ff2eaff85e744b361?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=celery).
> Report is 1 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #900      +/-   ##
==========================================
- Coverage   88.24%   87.67%   -0.57%     
==========================================
  Files          32       32              
  Lines        1012     1014       +2     
  Branches      105       82      -23     
==========================================
- Hits          893      889       -4     
- Misses        101      107       +6     
  Partials       18       18              
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/celery/django-celery-beat/pull/900?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=celery).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=celery).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: auvipy
created_at: 2025-05-17T08:28:23Z
body:  
  =================================== FAILURES ===================================
  ___________ test_DatabaseScheduler.test_crontab_timezone_conversion ____________
  
  self = <t.unit.test_schedulers.test_DatabaseScheduler object at 0x7f2fbbbd74d0>
  mock_get_tz = <MagicMock name='get_current_timezone' id='139842991386848'>
  mock_aware_now = <MagicMock name='aware_now' id='139842991387184'>
  
      @pytest.mark.django_db
      @patch('django_celery_beat.schedulers.aware_now')
      @patch('django.utils.timezone.get_current_timezone')
      def test_crontab_timezone_conversion(self, mock_get_tz, mock_aware_now):
          # Set up mocks for server timezone and current time
          from datetime import datetime
          server_tz = ZoneInfo("Asia/Tokyo")
      
          mock_get_tz.return_value = server_tz
      
          # Server time is 17:00 Tokyo time
          mock_now_dt = datetime(2023, 1, 1, 17, 0, 0, tzinfo=server_tz)
          mock_aware_now.return_value = mock_now_dt
      
          # Create tasks with the following crontab schedules:
          # 1. UTC task at hour 8 - equivalent to 17:00 Tokyo time (current hour)
          #    - should be included
          # 2. New York task at hour 3 - equivalent to 17:00 Tokyo time
          #    (current hour) - should be included
          # 3. UTC task at hour 3 - equivalent to 12:00 Tokyo time
          #    - should be excluded (outside window)
      
          # Create crontab schedules in different timezones
          utc_current_hour = CrontabSchedule.objects.create(
              hour='8', timezone='UTC'
          )
          ny_current_hour = CrontabSchedule.objects.create(
              hour='3', timezone='America/New_York'
          )
          utc_outside_window = CrontabSchedule.objects.create(
              hour='3', timezone='UTC'
          )
      
          # Create periodic tasks using these schedules
          task_utc_current = self.create_model(
              name='utc-current-hour',
              crontab=utc_current_hour
          )
          task_utc_current.save()
      
          task_ny_current = self.create_model(
              name='ny-current-hour',
              crontab=ny_current_hour
          )
          task_ny_current.save()
      
          task_utc_outside = self.create_model(
              name='utc-outside-window',
              crontab=utc_outside_window
          )
          task_utc_outside.save()
      
          # Run the scheduler's exclusion logic
          exclude_query = self.s._get_crontab_exclude_query()
      
          # Get excluded task IDs
          excluded_tasks = set(
              PeriodicTask.objects.filter(exclude_query).values_list(
                  'id', flat=True
              )
          )
      
          # Current hour tasks in different timezones should not be excluded
          assert task_utc_current.id not in excluded_tasks, (
              "UTC current hour task should be included"
          )
          assert task_ny_current.id not in excluded_tasks, (
              "New York current hour task should be included"
          )
      
          # Task outside window should be excluded
  >       assert task_utc_outside.id in excluded_tasks, (
              "UTC outside window task should be excluded"
          )
  E       AssertionError: UTC outside window task should be excluded
  E       assert 15 in {11}
  E        +  where 15 = <PeriodicTask: utc-outside-window: * 3 * * * (m/h/dM/MY/d) UTC>.id
  
  t/unit/test_schedulers.py:1133: AssertionError
author_association: MEMBER
user: login: alirafiei75
created_at: 2025-05-17T08:31:06Z
body: yes I will. the copilot suggestion was wrong
author_association: CONTRIBUTOR
user: login: alirafiei75
created_at: 2025-05-17T08:39:45Z
body: As we need both padded and non-padded hours ("1" and "01"), we cannot do it in just one list comprehension, or if we do it would not be readable i think
author_association: CONTRIBUTOR
user: login: auvipy
created_at: 2025-05-17T08:40:25Z
body: can you apply the improvement and also add unit tests to verify this please?
author_association: MEMBER
user: login: auvipy
created_at: 2025-05-17T10:15:02Z
body: Yup, that looks better
author_association: MEMBER