Repository: dimagi/commcare-hq
Pull request #36186: Reset flag on DataSourceConfig
Author: login: kaapstorm
Labels: name: product/invisible
description: Change has no end-user visible impact
Created: 2025-04-09T21:27:18Z | Merged: 2025-04-18T15:24:24Z

## Technical Summary

Context:
* [Data Source rebuilding getting stuck for indefinite time after adding new Case Properties/Columns](https://dimagi.atlassian.net/browse/QA-7573)
* [Reset Data source build/rebuild flag when migrated instead of rebuild](https://dimagi.atlassian.net/browse/SC-4399)

This change determines which rebuild flags to set on `DataSourceConfiguration.meta.build`, based on whether the data source will automatically be rebuilt.

:fish: :tropical_fish: :blowfish: 

## Safety Assurance

### Safety story

* Tested locally:
  + Creating a new data source
  + Making changes that require rebuilds
  + Making changes that do not require rebuilds
  + Making changes when a rebuild is pending

### Automated test coverage

Yes.

### QA Plan

No.

### Rollback instructions

- [x] This PR can be reverted after deploy with no further considerations

### Labels & Review
- [x] Risk label is set correctly
- [x] The set of people pinged as reviewers is appropriate for the level of risk of the change



Diff:
diff --git a/corehq/apps/hqwebapp/tests/data/bootstrap5_diffs/userreports/edit_data_source.html.diff.txt b/corehq/apps/hqwebapp/tests/data/bootstrap5_diffs/userreports/edit_data_source.html.diff.txt
index 3c7a8d5ce9d5..fe717288f518 100644
--- a/corehq/apps/hqwebapp/tests/data/bootstrap5_diffs/userreports/edit_data_source.html.diff.txt
+++ b/corehq/apps/hqwebapp/tests/data/bootstrap5_diffs/userreports/edit_data_source.html.diff.txt
@@ -11,7 +11,22 @@
  
  {% block page_content %}
    {% if data_source.meta.build.is_rebuild_in_progress %}
-@@ -24,12 +24,12 @@
+@@ -12,24 +12,24 @@
+         {% if data_source.rebuild_failed %}
+           {% trans "Populating your report did not complete successfully." %}
+         {% elif data_source.meta.build.is_rebuilding %}
+-          {% trans "Please note that this data source is being rebuilt." %}
++          {% trans "Please note that this datasource is being rebuilt." %}
+         {% elif data_source.meta.build.is_rebuilding_in_place %}
+-          {% trans "Please note that this data source is being rebuilt in place." %}
++          {% trans "Please note that this datasource is being rebuilt in place." %}
+         {% endif %}
+       </h4>
+     </div>
+   {% elif data_source.meta.build.awaiting %}
+     <div class="alert alert-info">
+-      {% trans "Please note that this data source is yet to be built / rebuilt by CommCare HQ." %}
++      {% trans "Please note that this datasource is yet be built / rebuilt by CommCare HQ." %}
      </div>
    {% endif %}
    {% if data_source.get_id %}
@@ -135,8 +150,12 @@
    </ul>
  
    <div class="tab-content">
-@@ -241,7 +241,7 @@
-           {% trans "This datasource is read only, any changes made can not be saved." %}
+@@ -238,10 +238,10 @@
+     <div class="tab-pane fade in active" id="tabs-configuration">
+       {% if read_only %}
+         <div class="alert alert-info">
+-          {% trans "This data source is read only, any changes made can not be saved." %}
++          {% trans "This datasource is read only, any changes made can not be saved." %}
          </div>
        {% endif %}
 -      {% crispy form %}
@@ -144,6 +163,15 @@
      </div>
      <div class="tab-pane fade" id="tabs-usage">
        {% if not used_by_reports %}
+@@ -249,7 +249,7 @@
+           {% trans "Datasource currently unused" %}
+         </div>
+       {% else %}
+-        <p>{% trans "Reports dependent on this data source" %}</p>
++        <p>{% trans "Reports dependent on this datasource" %}</p>
+         <ul>
+           {% for report in used_by_reports %}
+             <li>
 @@ -276,8 +276,8 @@
                <div class="modal-header">
                  <button
diff --git a/corehq/apps/userreports/models.py b/corehq/apps/userreports/models.py
index b327517059e5..c17f951adf9a 100644
--- a/corehq/apps/userreports/models.py
+++ b/corehq/apps/userreports/models.py
@@ -18,7 +18,7 @@
 import requests
 import yaml
 from celery.states import FAILURE
-from couchdbkit.exceptions import BadValueError
+from couchdbkit.exceptions import BadValueError, ResourceConflict
 from django_bulk_update.helper import bulk_update as bulk_update_helper
 from jsonpath_ng.ext import parser
 from memoized import memoized
@@ -52,11 +52,17 @@
 )
 from corehq.apps.domain.models import AllowedUCRExpressionSettings
 from corehq.apps.registry.helper import DataRegistryHelper
-from corehq.apps.userreports.app_manager.data_source_meta import (
+from corehq.pillows.utils import get_deleted_doc_types
+from corehq.sql_db.connections import UCR_ENGINE_ID, connection_manager
+from corehq.util.couch import DocumentNotFound, get_document_or_not_found
+from corehq.util.quickcache import quickcache
+
+from .alembic_diffs import get_tables_to_rebuild
+from .app_manager.data_source_meta import (
     REPORT_BUILDER_DATA_SOURCE_TYPE_VALUES,
 )
-from corehq.apps.userreports.columns import get_expanded_column_config
-from corehq.apps.userreports.const import (
+from .columns import get_expanded_column_config
+from .const import (
     ALL_EXPRESSION_TYPES,
     DATA_SOURCE_TYPE_AGGREGATE,
     DATA_SOURCE_TYPE_STANDARD,
@@ -66,7 +72,7 @@
     UCR_SQL_BACKEND,
     VALID_REFERENCED_DOC_TYPES,
 )
-from corehq.apps.userreports.dbaccessors import (
+from .dbaccessors import (
     get_all_registry_data_source_ids,
     get_datasources_for_domain,
     get_number_of_registry_report_configs_by_data_source,
@@ -75,7 +81,7 @@
     get_registry_report_configs_for_domain,
     get_report_configs_for_domain,
 )
-from corehq.apps.userreports.exceptions import (
+from .exceptions import (
     BadSpecError,
     DataSourceConfigurationNotFoundError,
     DuplicateColumnIdError,
@@ -84,32 +90,31 @@
     StaticDataSourceConfigurationNotFoundError,
     ValidationError,
 )
-from corehq.apps.userreports.expressions.factory import ExpressionFactory
-from corehq.apps.userreports.extension_points import (
+from .expressions.factory import ExpressionFactory
+from .extension_points import (
     static_ucr_data_source_paths,
     static_ucr_report_paths,
 )
-from corehq.apps.userreports.filters.factory import FilterFactory
-from corehq.apps.userreports.indicators import CompoundIndicator
-from corehq.apps.userreports.indicators.factory import IndicatorFactory
-from corehq.apps.userreports.reports.factory import (
+from .filters.factory import FilterFactory
+from .indicators import CompoundIndicator
+from .indicators.factory import IndicatorFactory
+from .reports.factory import (
     ChartFactory,
     ReportColumnFactory,
     ReportOrderByFactory,
 )
-from corehq.apps.userreports.reports.filters.factory import ReportFilterFactory
-from corehq.apps.userreports.reports.filters.specs import FilterSpec
-from corehq.apps.userreports.specs import EvaluationContext, FactoryContext
-from corehq.apps.userreports.sql.util import decode_column_name
-from corehq.apps.userreports.util import (
+from .reports.filters.factory import ReportFilterFactory
+from .reports.filters.specs import FilterSpec
+from .specs import EvaluationContext, FactoryContext
+from .sql import get_metadata
+from .sql.util import decode_column_name
+from .util import (
     get_async_indicator_modify_lock_key,
     get_indicator_adapter,
+    get_table_name,
+    get_ucr_datasource_config_by_id,
     wrap_report_config_by_type,
 )
-from corehq.pillows.utils import get_deleted_doc_types
-from corehq.sql_db.connections import UCR_ENGINE_ID, connection_manager
-from corehq.util.couch import DocumentNotFound, get_document_or_not_found
-from corehq.util.quickcache import quickcache
 
 ID_REGEX_CHECK = re.compile(r"^[\w\-:]+$")
 
@@ -744,15 +749,87 @@ def pk_columns(self):
 
     @cached_property
     def rebuild_failed(self):
-        # `has_died()` returns `None` if we can't use the Flower API,
-        # and so we don't know. Treating `None` as falsy allows calling
-        # code to give `rebuild_has_died` the benefit of the doubt.
+        # `DataSourceBuildInformation.rebuild_failed()` returns `None`
+        # if we don't know whether the rebuild has failed. Calling code
+        # can give the benefit of the doubt by treating `None` as falsy.
         return self.meta.build.rebuild_failed(self._id)
 
     @property
     def rebuild_awaiting_or_in_progress(self):
         return self.meta.build.awaiting or (self.meta.build.is_rebuild_in_progress and not self.rebuild_failed)
 
+    def set_rebuild_flags(self):
+        """
+        Sets rebuild flags based on whether a build is required.
+
+        If a build is in progress, and diffs require a rebuild, then the
+        awaiting flag is set. If a build is already awaiting, and diffs
+        do not require a rebuild, then the awaiting flag remains set.
+        """
+        from .rebuild import get_table_diffs
+
+        if not self.meta.build.awaiting:
+            engine = connection_manager.get_engine(self.engine_id)
+            table_name = get_table_name(self.domain, self.table_id)
+            engine_metadata = get_metadata(self.engine_id)
+            diffs = get_table_diffs(engine, [table_name], engine_metadata)
+            tables_to_rebuild = get_tables_to_rebuild(diffs)
+            if table_name in tables_to_rebuild:
+                self.set_build_queued()
+            else:
+                self.set_build_not_required()
+
+    def set_build_queued(self, *, reset_init_fin=True):
+        self.meta.build.awaiting = True
+        if reset_init_fin:
+            self.meta.build.initiated = None
+            self.meta.build.finished = False
+
+    def set_build_not_required(self):
+        self.meta.build.awaiting = False
+        self.meta.build.initiated = None
+        self.meta.build.finished = False
+
+    def save_build_started(self, *, in_place=False):
+        # Save the start time now in case anything goes wrong. This way
+        # we'll be able to see if the rebuild started a long time ago
+        # without finishing.
+        start_time = datetime.utcnow()
+        self.meta.build.awaiting = False
+        if in_place:
+            self.meta.build.initiated_in_place = start_time
+            self.meta.build.finished_in_place = False
+        else:
+            self.meta.build.initiated = start_time
+            self.meta.build.finished = False
+        self.meta.build.rebuilt_asynchronously = False
+        self.save()
+
+    def save_build_finished(self, *, in_place=False):
+        if in_place:
+            self.meta.build.finished_in_place = True
+        else:
+            self.meta.build.finished = True
+
+        try:
+            self.save()
+        except ResourceConflict:
+            current_config = get_ucr_datasource_config_by_id(self._id)
+            # check that a new build has not yet started
+            if in_place:
+                if (
+                    self.meta.build.initiated_in_place
+                    == current_config.meta.build.initiated_in_place
+                ):
+                    current_config.meta.build.finished_in_place = True
+            else:
+                if (
+                    self.meta.build.initiated
+                    == current_config.meta.build.initiated
+                ):
+                    current_config.meta.build.finished = True
+            current_config.save()
+
 
 class RegistryDataSourceConfiguration(DataSourceConfiguration):
     """This is a special data source that can contain data from
@@ -951,9 +1028,7 @@ def get_expanded_columns(self, column_config):
     @property
     @memoized
     def cached_data_source(self):
-        from corehq.apps.userreports.reports.data_source import (
-            ConfigurableReportDataSource,
-        )
+        from .reports.data_source import ConfigurableReportDataSource
         return ConfigurableReportDataSource.from_spec(self).data_source
 
     @property
@@ -1018,9 +1093,7 @@ def get_languages(self):
         return langs
 
     def validate(self, required=True):
-        from corehq.apps.userreports.reports.data_source import (
-            ConfigurableReportDataSource,
-        )
+        from .reports.data_source import ConfigurableReportDataSource
 
         def _check_for_duplicates(supposedly_unique_list, error_msg):
             # http://stackoverflow.com/questions/9835762/find-and-list-duplicates-in-python-list
diff --git a/corehq/apps/userreports/pillow.py b/corehq/apps/userreports/pillow.py
index 59039d27cd40..2df262380d23 100644
--- a/corehq/apps/userreports/pillow.py
+++ b/corehq/apps/userreports/pillow.py
@@ -6,35 +6,43 @@
 
 from django.conf import settings
 
+from pillowtop.checkpoints.manager import KafkaPillowCheckpoint
+from pillowtop.const import DEFAULT_PROCESSOR_CHUNK_SIZE
+from pillowtop.exceptions import PillowConfigError
+from pillowtop.logger import pillow_logging
+from pillowtop.pillow.interface import ConstructedPillow
+from pillowtop.processors import BulkPillowProcessor
+from pillowtop.utils import (
+    bulk_fetch_changes_docs,
+    ensure_document_exists,
+    ensure_matched_revisions,
+)
+
 from corehq.apps.change_feed.consumer.feed import (
     KafkaChangeFeed,
     KafkaCheckpointEventHandler,
 )
-from corehq.apps.change_feed.topics import LOCATION as LOCATION_TOPIC, CASE_TOPICS
+from corehq.apps.change_feed.topics import CASE_TOPICS
+from corehq.apps.change_feed.topics import LOCATION as LOCATION_TOPIC
 from corehq.apps.domain.dbaccessors import get_domain_ids_by_names
-from corehq.apps.domain_migration_flags.api import all_domains_with_migrations_in_progress
-from corehq.apps.userreports.const import KAFKA_TOPICS
-from corehq.apps.userreports.data_source_providers import (
-    DynamicDataSourceProvider,
-    StaticDataSourceProvider, RegistryDataSourceProvider,
-)
-from corehq.apps.userreports.exceptions import (
-    UserReportsWarning,
+from corehq.apps.domain_migration_flags.api import (
+    all_domains_with_migrations_in_progress,
 )
-from corehq.apps.userreports.models import AsyncIndicator
-from corehq.apps.userreports.pillow_utils import rebuild_sql_tables
-from corehq.apps.userreports.specs import EvaluationContext
-from corehq.apps.userreports.util import get_indicator_adapter
 from corehq.pillows.base import is_couch_change_for_sql_domain
 from corehq.util.metrics import metrics_counter, metrics_histogram_timer
 from corehq.util.timer import TimingContext
-from pillowtop.checkpoints.manager import KafkaPillowCheckpoint
-from pillowtop.const import DEFAULT_PROCESSOR_CHUNK_SIZE
-from pillowtop.exceptions import PillowConfigError
-from pillowtop.logger import pillow_logging
-from pillowtop.pillow.interface import ConstructedPillow
-from pillowtop.processors import BulkPillowProcessor
-from pillowtop.utils import ensure_document_exists, ensure_matched_revisions, bulk_fetch_changes_docs
+
+from .const import KAFKA_TOPICS
+from .data_source_providers import (
+    DynamicDataSourceProvider,
+    RegistryDataSourceProvider,
+    StaticDataSourceProvider,
+)
+from .exceptions import UserReportsWarning
+from .models import AsyncIndicator
+from .pillow_utils import rebuild_sql_tables
+from .specs import EvaluationContext
+from .util import get_indicator_adapter
 
 REBUILD_CHECK_INTERVAL = 3 * 60 * 60  # in seconds
 LONG_UCR_LOGGING_THRESHOLD = 0.5
@@ -460,7 +468,8 @@ def _process_chunk_for_domain(self, domain, changes_chunk):
                                     async_configs_by_doc_id[doc['_id']].append(adapter.config._id)
                                 else:
                                     try:
-                                        rows_to_save_by_adapter[adapter].extend(adapter.get_all_values(doc, eval_context))
+                                        rows = adapter.get_all_values(doc, eval_context)
+                                        rows_to_save_by_adapter[adapter].extend(rows)
                                     except Exception as e:
                                         change_exceptions.append((change, e))
                                     eval_context.reset_iteration()
@@ -759,8 +768,13 @@ def get_location_pillow(pillow_id='location-ucr-pillow', include_ucrs=None,
 
 def get_kafka_ucr_registry_pillow(
     pillow_id='kafka-ucr-registry',
-    num_processes=1, process_num=0, dedicated_migration_process=False,
-    processor_chunk_size=DEFAULT_PROCESSOR_CHUNK_SIZE, ucr_configs=None, **kwargs):
+    num_processes=1,
+    process_num=0,
+    dedicated_migration_process=False,
+    processor_chunk_size=DEFAULT_PROCESSOR_CHUNK_SIZE,
+    ucr_configs=None,
+    **kwargs,
+):
     """UCR pillow that reads from all 'case' Kafka topics and writes data into the UCR database tables
 
     Only UCRs backed by Data Registries are processed in this pillow.
diff --git a/corehq/apps/userreports/pillow_utils.py b/corehq/apps/userreports/pillow_utils.py
index 0339de257540..50bcb2bd23dd 100644
--- a/corehq/apps/userreports/pillow_utils.py
+++ b/corehq/apps/userreports/pillow_utils.py
@@ -1,13 +1,20 @@
 from collections import defaultdict
 
-from corehq.apps.userreports.exceptions import StaleRebuildError, TableRebuildError
 from couchdbkit import ResourceNotFound
-from corehq.apps.userreports.rebuild import migrate_tables, get_tables_rebuild_migrate, get_table_diffs
-from corehq.apps.userreports.sql import get_metadata
-from corehq.apps.userreports.tasks import rebuild_indicators
+
+from pillowtop.logger import pillow_logging
+
 from corehq.sql_db.connections import connection_manager
 from corehq.util.soft_assert import soft_assert
-from pillowtop.logger import pillow_logging
+
+from .exceptions import StaleRebuildError, TableRebuildError
+from .rebuild import (
+    get_table_diffs,
+    get_tables_rebuild_migrate,
+    migrate_tables,
+)
+from .sql import get_metadata
+from .tasks import rebuild_indicators
 
 
 def _is_datasource_active(adapter):
diff --git a/corehq/apps/userreports/sql/adapter.py b/corehq/apps/userreports/sql/adapter.py
index dcff13a22781..ee3d3b2c293f 100644
--- a/corehq/apps/userreports/sql/adapter.py
+++ b/corehq/apps/userreports/sql/adapter.py
@@ -6,21 +6,22 @@
 import psycopg2
 import sqlalchemy
 from memoized import memoized
-from sqlalchemy.exc import ProgrammingError, OperationalError
+from sqlalchemy.exc import OperationalError, ProgrammingError
 from sqlalchemy.ext.declarative import declarative_base
 from sqlalchemy.schema import Index, PrimaryKeyConstraint
 
-from corehq.apps.userreports.adapter import IndicatorAdapter
-from corehq.apps.userreports.exceptions import (
+from corehq.sql_db.connections import connection_manager
+from corehq.util.test_utils import unit_testing_only
+
+from ..adapter import IndicatorAdapter
+from ..exceptions import (
     ColumnNotFoundError,
     TableRebuildError,
     translate_programming_error,
 )
-from corehq.apps.userreports.sql.columns import column_to_sql
-from corehq.apps.userreports.util import get_table_name
-from corehq.sql_db.connections import connection_manager
-from corehq.util.test_utils import unit_testing_only
-from corehq.apps.userreports.util import register_data_source_row_change
+from ..util import get_table_name, register_data_source_row_change
+from .columns import column_to_sql
+
 logger = logging.getLogger(__name__)
 
 
@@ -349,7 +350,7 @@ def handle_exception(self, doc, exception):
         orig_exception = getattr(exception, 'orig', None)
         if orig_exception and isinstance(orig_exception, psycopg2.IntegrityError):
             if orig_exception.pgcode == psycopg2.errorcodes.NOT_NULL_VIOLATION:
-                from corehq.apps.userreports.models import InvalidUCRData
+                from ..models import InvalidUCRData
                 InvalidUCRData.objects.get_or_create(
                     doc_id=doc['_id'],
                     indicator_config_id=self.config._id,
diff --git a/corehq/apps/userreports/tasks.py b/corehq/apps/userreports/tasks.py
index 64a437b7a8fa..8a43de899a2f 100644
--- a/corehq/apps/userreports/tasks.py
+++ b/corehq/apps/userreports/tasks.py
@@ -13,7 +13,7 @@
     ProtocolError,
 )
 from celery.schedules import crontab
-from couchdbkit import ResourceConflict, ResourceNotFound
+from couchdbkit import ResourceNotFound
 
 from couchexport.models import Format
 from dimagi.utils.chunked import chunked
@@ -27,28 +27,6 @@
     get_document_store_for_doc_type,
 )
 from corehq.apps.reports.util import send_report_download_email
-from corehq.apps.userreports.const import (
-    ASYNC_INDICATOR_CHUNK_SIZE,
-    ASYNC_INDICATOR_MAX_RETRIES,
-    ASYNC_INDICATOR_QUEUE_TIME,
-    UCR_CELERY_QUEUE,
-    UCR_INDICATOR_CELERY_QUEUE,
-)
-from corehq.apps.userreports.exceptions import (
-    DataSourceConfigurationNotFoundError,
-)
-from corehq.apps.userreports.models import (
-    AsyncIndicator,
-    DataSourceActionLog,
-    id_is_static,
-)
-from corehq.apps.userreports.rebuild import DataSourceResumeHelper
-from corehq.apps.userreports.specs import EvaluationContext
-from corehq.apps.userreports.util import (
-    get_async_indicator_modify_lock_key,
-    get_indicator_adapter,
-    get_ucr_datasource_config_by_id,
-)
 from corehq.elastic import ESError
 from corehq.util.context_managers import notify_someone
 from corehq.util.decorators import serial_task
@@ -63,6 +41,23 @@
 from corehq.util.timer import TimingContext
 from corehq.util.view_utils import reverse
 
+from .const import (
+    ASYNC_INDICATOR_CHUNK_SIZE,
+    ASYNC_INDICATOR_MAX_RETRIES,
+    ASYNC_INDICATOR_QUEUE_TIME,
+    UCR_CELERY_QUEUE,
+    UCR_INDICATOR_CELERY_QUEUE,
+)
+from .exceptions import DataSourceConfigurationNotFoundError
+from .models import AsyncIndicator, DataSourceActionLog, id_is_static
+from .rebuild import DataSourceResumeHelper
+from .specs import EvaluationContext
+from .util import (
+    get_async_indicator_modify_lock_key,
+    get_indicator_adapter,
+    get_ucr_datasource_config_by_id,
+)
+
 celery_task_logger = logging.getLogger('celery.task')
 
 
@@ -111,13 +106,7 @@ def rebuild_indicators(
                 raise AssertionError("Engine ID does not match adapter")
 
         if not id_is_static(indicator_config_id):
-            # Save the start time now in case anything goes wrong. This way we'll be
-            # able to see if the rebuild started a long time ago without finishing.
-            config.meta.build.awaiting = False
-            config.meta.build.initiated = datetime.utcnow()
-            config.meta.build.finished = False
-            config.meta.build.rebuilt_asynchronously = False
-            config.save()
+            config.save_build_started()
 
         skip_log = bool(limit > 0)  # don't store log for temporary report builder UCRs
         rows_count_before_rebuild = _get_rows_count_from_existing_table(adapter)
@@ -142,11 +131,7 @@ def rebuild_indicators_in_place(indicator_config_id, initiated_by=None, source=N
     with notify_someone(initiated_by, success_message=success, error_message=failure, send=True):
         adapter = get_indicator_adapter(config)
         if not id_is_static(indicator_config_id):
-            config.meta.build.awaiting = False
-            config.meta.build.initiated_in_place = datetime.utcnow()
-            config.meta.build.finished_in_place = False
-            config.meta.build.rebuilt_asynchronously = False
-            config.save()
+            config.save_build_started(in_place=True)
 
         rows_count_before_rebuild = _get_rows_count_from_existing_table(adapter)
         try:
@@ -194,7 +179,7 @@ def _report_metric_number_of_days_since_first_build(config, action):
 
 
 def _report_metric_rebuild_error(config, action):
-    from corehq.apps.userreports.views import number_of_records_to_be_processed
+    from .views import number_of_records_to_be_processed
     expected_rows_to_process = number_of_records_to_be_processed(config)
     metrics_gauge(
         f'commcare.ucr.{action}.failed.expected_rows_to_process',
@@ -265,27 +250,12 @@ def _iteratively_build_table(config, resume_helper=None, in_place=False, limit=-
 
     resume_helper.clear_resume_info()
     if not id_is_static(indicator_config_id):
-        if in_place:
-            config.meta.build.finished_in_place = True
-        else:
-            config.meta.build.finished = True
-        try:
-            config.save()
-        except ResourceConflict:
-            current_config = get_ucr_datasource_config_by_id(config._id)
-            # check that a new build has not yet started
-            if in_place:
-                if config.meta.build.initiated_in_place == current_config.meta.build.initiated_in_place:
-                    current_config.meta.build.finished_in_place = True
-            else:
-                if config.meta.build.initiated == current_config.meta.build.initiated:
-                    current_config.meta.build.finished = True
-            current_config.save()
+        config.save_build_finished(in_place=in_place)
 
 
 @task(serializer='pickle', queue=UCR_CELERY_QUEUE, ignore_result=True)
 def delete_data_source_task(domain, config_id):
-    from corehq.apps.userreports.views import delete_data_source_shared
+    from .views import delete_data_source_shared
     delete_data_source_shared(domain, config_id)
 
 
diff --git a/corehq/apps/userreports/templates/userreports/bootstrap3/edit_data_source.html b/corehq/apps/userreports/templates/userreports/bootstrap3/edit_data_source.html
index 7d21102f9d47..8883fb46b5c0 100644
--- a/corehq/apps/userreports/templates/userreports/bootstrap3/edit_data_source.html
+++ b/corehq/apps/userreports/templates/userreports/bootstrap3/edit_data_source.html
@@ -12,15 +12,15 @@ <h4>
         {% if data_source.rebuild_failed %}
           {% trans "Populating your report did not complete successfully." %}
         {% elif data_source.meta.build.is_rebuilding %}
-          {% trans "Please note that this datasource is being rebuilt." %}
+          {% trans "Please note that this data source is being rebuilt." %}
         {% elif data_source.meta.build.is_rebuilding_in_place %}
-          {% trans "Please note that this datasource is being rebuilt in place." %}
+          {% trans "Please note that this data source is being rebuilt in place." %}
         {% endif %}
       </h4>
     </div>
   {% elif data_source.meta.build.awaiting %}
     <div class="alert alert-info">
-      {% trans "Please note that this datasource is yet be built / rebuilt by CommCare HQ." %}
+      {% trans "Please note that this data source is yet to be built / rebuilt by CommCare HQ." %}
     </div>
   {% endif %}
   {% if data_source.get_id %}
@@ -238,7 +238,7 @@ <h4>
     <div class="tab-pane fade in active" id="tabs-configuration">
       {% if read_only %}
         <div class="alert alert-info">
-          {% trans "This datasource is read only, any changes made can not be saved." %}
+          {% trans "This data source is read only, any changes made can not be saved." %}
         </div>
       {% endif %}
       {% crispy form %}
@@ -249,7 +249,7 @@ <h4>
           {% trans "Datasource currently unused" %}
         </div>
       {% else %}
-        <p>{% trans "Reports dependent on this datasource" %}</p>
+        <p>{% trans "Reports dependent on this data source" %}</p>
         <ul>
           {% for report in used_by_reports %}
             <li>
diff --git a/corehq/apps/userreports/tests/test_data_source_config.py b/corehq/apps/userreports/tests/test_data_source_config.py
index 59491338b743..06c0076ddf1e 100644
--- a/corehq/apps/userreports/tests/test_data_source_config.py
+++ b/corehq/apps/userreports/tests/test_data_source_config.py
@@ -7,23 +7,17 @@
 from jsonobject.exceptions import BadValueError
 
 from corehq.apps.domain.models import AllowedUCRExpressionSettings
-from corehq.apps.userreports.const import (
-    UCR_NAMED_EXPRESSION,
-    UCR_NAMED_FILTER,
-)
-from corehq.apps.userreports.exceptions import BadSpecError
-from corehq.apps.userreports.models import (
-    DataSourceConfiguration,
-    UCRExpression,
-)
-from corehq.apps.userreports.specs import EvaluationContext, FactoryContext
-from corehq.apps.userreports.tests.utils import (
-    get_sample_data_source,
-    get_sample_doc_and_indicators,
-)
 from corehq.sql_db.connections import UCR_ENGINE_ID
 from corehq.util.test_utils import flag_enabled
 
+from ..alembic_diffs import DiffTypes
+from ..const import UCR_NAMED_EXPRESSION, UCR_NAMED_FILTER
+from ..exceptions import BadSpecError
+from ..models import DataSourceConfiguration, UCRExpression
+from ..specs import EvaluationContext, FactoryContext
+from ..tests.utils import get_sample_data_source, get_sample_doc_and_indicators
+from ..util import get_table_name
+
 
 class TestDataSourceConfigAllowedExpressionsValidation(TestCase):
 
@@ -341,6 +335,21 @@ def test_invalid_expression(self):
 
 class DataSourceConfigurationTests(TestCase):
 
+    @classmethod
+    def setUpClass(cls):
+        super().setUpClass()
+        for domain, table_id in [
+            ('foo', 'foo1'),
+            ('foo', 'foo2'),
+            ('bar', 'bar1')
+        ]:
+            config = DataSourceConfiguration(
+                domain=domain,
+                table_id=table_id,
+                referenced_doc_type='XFormInstance')
+            config.save()
+            cls.addClassCleanup(config.delete)
+
     def test_by_domain_returns_relevant_datasource_configs(self):
         results = DataSourceConfiguration.by_domain('foo')
         self.assertEqual(len(results), 2)
@@ -385,17 +394,90 @@ def test_create_requires_doc_type(self):
         with self.assertRaises(BadValueError):
             DataSourceConfiguration(domain='domain', table_id='table').save()
 
-    @classmethod
-    def setUpClass(cls):
-        super().setUpClass()
-        for domain, table_id in [('foo', 'foo1'), ('foo', 'foo2'),
-                                 ('bar', 'bar1')]:
-            config = DataSourceConfiguration(
-                domain=domain,
-                table_id=table_id,
-                referenced_doc_type='XFormInstance')
-            config.save()
-            cls.addClassCleanup(config.delete)
+
+@patch('corehq.apps.userreports.models.DataSourceBuildInformation.rebuild_failed',
+       MagicMock(return_value=None))
+class DataSourceConfigurationRebuildTests(TestCase):
+
+    def setUp(self):
+        self.config = DataSourceConfiguration(
+            domain='test-domain',
+            table_id='table_id',
+            referenced_doc_type='XFormInstance'
+        )
+        self.config.save()
+        self.addCleanup(self.config.delete)
+
+    def test_rebuild_failed(self):
+        assert self.config.rebuild_failed is None
+
+    def test_set_build_not_required(self):
+        self.config.set_build_not_required()
+        assert self.config.meta.build.awaiting is False
+        assert self.config.meta.build.initiated is None
+        assert self.config.meta.build.finished is False
+
+    def test_set_build_queued(self):
+        self.config.set_build_queued()
+        assert self.config.meta.build.awaiting is True
+        assert self.config.meta.build.initiated is None
+        assert self.config.meta.build.finished is False
+
+    def test_build_not_required(self):
+        self.config.set_build_not_required()
+        assert self.config.rebuild_awaiting_or_in_progress is None
+
+    def test_build_queued(self):
+        self.config.set_build_queued()
+        assert self.config.rebuild_awaiting_or_in_progress is True
+
+    def test_build_initiated(self):
+        self.config.meta.build.awaiting = False
+        self.config.meta.build.initiated = datetime.datetime.now()
+        self.config.meta.build.finished = False
+        assert self.config.rebuild_awaiting_or_in_progress is True
+
+    def test_build_finished(self):
+        self.config.meta.build.awaiting = False
+        self.config.meta.build.initiated = datetime.datetime.now()
+        self.config.meta.build.finished = True
+        assert self.config.rebuild_awaiting_or_in_progress is None
+
+    @patch('corehq.apps.userreports.rebuild.get_table_diffs')
+    def test_set_rebuild_flags_queued(self, mock_get_table_diffs):
+        table_name = get_table_name(self.config.domain, self.config.table_id)
+        diff = MagicMock(table_name=table_name, type=DiffTypes.MODIFY_TYPE)
+        mock_get_table_diffs.return_value = [diff]
+        self.config.set_rebuild_flags()
+        assert self.config.meta.build.awaiting is True
+
+    @patch('corehq.apps.userreports.rebuild.get_table_diffs')
+    def test_set_rebuild_flags_not_required(self, mock_get_table_diffs):
+        mock_get_table_diffs.return_value = []
+        self.config.set_rebuild_flags()
+        assert self.config.meta.build.awaiting is False
+
+    @patch('corehq.apps.userreports.rebuild.get_table_diffs')
+    def test_set_rebuild_flags_not_required_awaiting(self, mock_get_table_diffs):
+        # If a rebuild is already awaiting, it is not unset.
+        mock_get_table_diffs.return_value = []
+        self.config.meta.build.awaiting = True
+        self.config.set_rebuild_flags()
+        assert self.config.meta.build.awaiting is True
+
+    @patch('corehq.apps.userreports.rebuild.get_table_diffs')
+    def test_set_rebuild_flags_in_progress(self, mock_get_table_diffs):
+        # If a rebuild is in progress, it is set to awaiting.
+        table_name = get_table_name(self.config.domain, self.config.table_id)
+        diff = MagicMock(table_name=table_name, type=DiffTypes.MODIFY_TYPE)
+        mock_get_table_diffs.return_value = [diff]
+
+        self.config.meta.build.awaiting = False
+        self.config.meta.build.initiated = datetime.datetime.now()
+        self.config.meta.build.finished = False
+
+        self.config.set_rebuild_flags()
+        assert self.config.meta.build.awaiting is True
 
 
 @patch('corehq.apps.userreports.models.AllowedUCRExpressionSettings.disallowed_ucr_expressions',
diff --git a/corehq/apps/userreports/ui/forms.py b/corehq/apps/userreports/ui/forms.py
index db9ab21e4367..04500ac5d427 100644
--- a/corehq/apps/userreports/ui/forms.py
+++ b/corehq/apps/userreports/ui/forms.py
@@ -13,13 +13,14 @@
 from corehq.apps.domain.models import AllowedUCRExpressionSettings
 from corehq.apps.hqwebapp import crispy as hqcrispy
 from corehq.apps.hqwebapp.widgets import BootstrapCheckboxInput
-from corehq.apps.userreports.const import DATA_SOURCE_REBUILD_RESTRICTED_AT
-from corehq.apps.userreports.models import guess_data_source_type
-from corehq.apps.userreports.ui import help_text
-from corehq.apps.userreports.ui.fields import JsonField, ReportDataSourceField
-from corehq.apps.userreports.util import get_table_name
 from corehq.util.metrics import metrics_counter
 
+from ..const import DATA_SOURCE_REBUILD_RESTRICTED_AT
+from ..models import guess_data_source_type
+from ..util import get_table_name
+from . import help_text
+from .fields import JsonField, ReportDataSourceField
+
 
 class DocumentFormBase(forms.Form):
     """
@@ -179,7 +180,7 @@ class ConfigurableDataSourceEditForm(DocumentFormBase):
         required=False,
         label="",
         widget=forms.HiddenInput(),
-        help_text=help_text.ANALYTICS
+        help_text=help_text.COMMCARE_ANALYTICS
     )
 
     def __init__(self, domain, data_source_config, read_only, *args, **kwargs):
@@ -287,9 +288,7 @@ def _check_for_asynchronous(self, config):
                 )
 
     def save(self, commit=False):
-        self.instance.meta.build.finished = False
-        self.instance.meta.build.initiated = None
-        self.instance.meta.build.awaiting = True
+        self.instance.set_rebuild_flags()
         instance = super(ConfigurableDataSourceEditForm, self).save(commit)
         self._report_edit_datasource_metrics()
         return instance
diff --git a/corehq/apps/userreports/ui/help_text.py b/corehq/apps/userreports/ui/help_text.py
index a2820d0169a9..69a3d30e25d1 100644
--- a/corehq/apps/userreports/ui/help_text.py
+++ b/corehq/apps/userreports/ui/help_text.py
@@ -41,6 +41,7 @@
     'wherever an expression goes using: <code>{"type": "named", "name": "myvarname"}</code>')
 NAMED_FILTER = _('These behave exactly like named expressions (see above), except the values '
                  'should be a valid filter, and they can be used wherever filters are used above.')
-ANALYTICS = _(
-    'Enabling this will let this data source and the data be imported into Analytics'
+COMMCARE_ANALYTICS = _(
+    'Enabling this will let this data source and the data be imported into '
+    'CommCare Analytics'
 )
diff --git a/corehq/apps/userreports/views.py b/corehq/apps/userreports/views.py
index 40f5b15e8c67..601ce4bae9f6 100644
--- a/corehq/apps/userreports/views.py
+++ b/corehq/apps/userreports/views.py
@@ -1204,12 +1204,20 @@ def post(self, request, *args, **kwargs):
         try:
             if self.edit_form.is_valid():
                 config = self.edit_form.save(commit=True)
-                messages.success(request, _('Data source "{}" saved!').format(
+                message_list = [_('Data source "{}" saved.').format(
                     config.display_name
-                ))
-                messages.success(request, _(
-                    'This data source will be built / rebuilt automatically by CommCare HQ.'
-                ))
+                )]
+                if config.meta.build.awaiting:
+                    message_list.append(_(
+                        'This data source will be built / rebuilt automatically by CommCare HQ.'
+                    ))
+                else:
+                    message_list.append(_(
+                        'CommCare HQ will not perform a data source rebuild '
+                        'for the changes made. Please initiate one manually '
+                        'if needed.'
+                    ))
+                messages.success(request, ' '.join(message_list))
                 if self.config_id is None:
                     return HttpResponseRedirect(reverse(
                         EditDataSourceView.urlname, args=[self.domain, config._id])
@@ -1331,9 +1339,8 @@ def undelete_data_source(request, domain, config_id):
 def rebuild_data_source(request, domain, config_id):
     config, is_static = get_datasource_config_or_404(config_id, domain)
 
-    response = _redirect_response_if_build_awaiting(request, domain, config)
-    if response:
-        return response
+    if config.meta.build.awaiting:
+        return _build_awaiting_redirect_response(request, domain, config_id)
 
     if not config.asynchronous and toggles.RESTRICT_DATA_SOURCE_REBUILD.enabled(domain):
         number_of_records = number_of_records_to_be_processed(datasource_configuration=config)
@@ -1362,21 +1369,20 @@ def rebuild_data_source(request, domain, config_id):
     ))
 
 
-def _redirect_response_if_build_awaiting(request, domain, config):
-    if config.meta.build.awaiting:
-        messages.error(
-            request,
-            _('Rebuilding is not available until CommCare HQ finishes building / rebuilding.')
-        )
-        return HttpResponseRedirect(reverse(
-            EditDataSourceView.urlname, args=[domain, config.get_id]
-        ))
+def _build_awaiting_redirect_response(request, domain, config_id):
+    messages.error(
+        request,
+        _('Rebuilding is not available until CommCare HQ finishes building / rebuilding.')
+    )
+    return HttpResponseRedirect(reverse(
+        EditDataSourceView.urlname, args=[domain, config_id]
+    ))
 
 
 def _prep_data_source_for_rebuild(data_source_config, is_static):
     save_config = False
     if not is_static:
-        data_source_config.meta.build.awaiting = True
+        data_source_config.set_build_queued(reset_init_fin=False)
         save_config = True
     if data_source_config.is_deactivated:
         data_source_config.is_deactivated = False
@@ -1477,7 +1483,7 @@ def resume_building_data_source(request, domain, config_id):
         )
     else:
         if not is_static:
-            config.meta.build.awaiting = True
+            config.set_build_queued(reset_init_fin=False)
             config.save()
         messages.success(
             request,
@@ -1494,9 +1500,8 @@ def resume_building_data_source(request, domain, config_id):
 def build_data_source_in_place(request, domain, config_id):
     config, is_static = get_datasource_config_or_404(config_id, domain)
 
-    response = _redirect_response_if_build_awaiting(request, domain, config)
-    if response:
-        return response
+    if config.meta.build.awaiting:
+        return _build_awaiting_redirect_response(request, domain, config_id)
 
     _prep_data_source_for_rebuild(config, is_static)
 


Reviews:
user: login: mkangia
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T09:44:19Z
user: login: mkangia
body: Nice clean up @kaapstorm :+1: 
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T09:44:34Z
user: login: kaapstorm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T12:27:23Z
user: login: mkangia
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T22:11:07Z
user: login: kaapstorm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T11:30:16Z
user: login: mkangia
body: Sorry I missed something earlier, so just confirming on that, one commit might not be needed.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T10:58:36Z
user: login: kaapstorm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T12:55:07Z
user: login: kaapstorm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T13:12:15Z
user: login: mkangia
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T13:34:48Z
user: login: mkangia
body: Thanks @kaapstorm 
This looks great. Just couple of minor things.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T11:56:01Z
user: login: kaapstorm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T12:28:12Z
user: login: kaapstorm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T12:28:22Z
user: login: kaapstorm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T12:28:25Z
user: login: mkangia
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T15:24:04Z
user: login: mkangia
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T15:24:23Z

Review comments:
diff_hunk: @@ -103,6 +103,10 @@ def rebuild_table(adapter, diffs=None):
         adapter.log_table_rebuild_skipped(source='pillowtop', diffs=diff_dicts)
         return
 
+    if not config.is_static:
+        config.set_build_queued(reset_init_fin=False)
path: corehq/apps/userreports/pillow_utils.py
user: login: mkangia
body: and we will reset awaiting to False if the table is being migrated and not rebuilt by pillows?
created_at: 2025-04-10T09:44:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -103,6 +103,10 @@ def rebuild_table(adapter, diffs=None):
         adapter.log_table_rebuild_skipped(source='pillowtop', diffs=diff_dicts)
         return
 
+    if not config.is_static:
+        config.set_build_queued(reset_init_fin=False)
path: corehq/apps/userreports/pillow_utils.py
user: login: kaapstorm
body: Ah! Thank you, this is the feedback I'm looking for.

If the table is migrated and not rebuilt, are these the correct flags? 
```python
    config.meta.build.awaiting = False

    # Reset these, or leave unchanged?
    config.meta.build.initiated = None
    config.meta.build.finished = False  # Or `True`?
```


created_at: 2025-04-11T12:27:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -103,6 +103,10 @@ def rebuild_table(adapter, diffs=None):
         adapter.log_table_rebuild_skipped(source='pillowtop', diffs=diff_dicts)
         return
 
+    if not config.is_static:
+        config.set_build_queued(reset_init_fin=False)
path: corehq/apps/userreports/pillow_utils.py
user: login: mkangia
body: I can confirm that this is correct and is the one that blocks users from taking more action.
`config.meta.build.awaiting = False`

I am not sure how and where others are relevant.
created_at: 2025-04-11T22:11:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -103,6 +103,10 @@ def rebuild_table(adapter, diffs=None):
         adapter.log_table_rebuild_skipped(source='pillowtop', diffs=diff_dicts)
         return
 
+    if not config.is_static:
+        config.set_build_queued(reset_init_fin=False)
path: corehq/apps/userreports/pillow_utils.py
user: login: kaapstorm
body: 0ebcf5dbec9
created_at: 2025-04-14T11:30:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -753,6 +754,52 @@ def rebuild_failed(self):
     def rebuild_awaiting_or_in_progress(self):
         return self.meta.build.awaiting or (self.meta.build.is_rebuild_in_progress and not self.rebuild_failed)
 
+    def set_build_queued(self, *, reset_init_fin=True):
+        self.meta.build.awaiting = True
+        if reset_init_fin:
+            self.meta.build.initiated = None
+            self.meta.build.finished = False
path: corehq/apps/userreports/models.py
user: login: mkangia
body: I noticed this only yesterday when looking at some code,
does this need to also reset initiated_in_place & finished_in_place?
I believe those are just the "in_place" equivalents when "Rebuild in place" is used, and I believe `set_build_queued` will be called for that operation too.
created_at: 2025-04-16T10:42:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -753,6 +754,52 @@ def rebuild_failed(self):
     def rebuild_awaiting_or_in_progress(self):
         return self.meta.build.awaiting or (self.meta.build.is_rebuild_in_progress and not self.rebuild_failed)
 
+    def set_build_queued(self, *, reset_init_fin=True):
+        self.meta.build.awaiting = True
+        if reset_init_fin:
+            self.meta.build.initiated = None
+            self.meta.build.finished = False
+
+    def save_build_started(self, *, in_place=False):
path: corehq/apps/userreports/models.py
user: login: mkangia
body: nit: I am confused by the asterik here. Is it meant to absorb any arguments?
created_at: 2025-04-16T10:45:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -753,6 +754,52 @@ def rebuild_failed(self):
     def rebuild_awaiting_or_in_progress(self):
         return self.meta.build.awaiting or (self.meta.build.is_rebuild_in_progress and not self.rebuild_failed)
 
+    def set_build_queued(self, *, reset_init_fin=True):
+        self.meta.build.awaiting = True
+        if reset_init_fin:
+            self.meta.build.initiated = None
+            self.meta.build.finished = False
path: corehq/apps/userreports/models.py
user: login: mkangia
body: Okay, reading more it seems you just retained the implementation as it was.
so, its possible rebuild in place has a bug?
created_at: 2025-04-16T10:47:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -103,6 +103,10 @@ def rebuild_table(adapter, diffs=None):
         adapter.log_table_rebuild_skipped(source='pillowtop', diffs=diff_dicts)
         return
 
+    if not config.is_static:
+        config.set_build_queued(reset_init_fin=False)
path: corehq/apps/userreports/pillow_utils.py
user: login: mkangia
body: Sorry I missed this earlier,
I believe at this point, `awaiting` is already set to True since that is done in the view so we should not need to queue it again here. The bug was that the flag was not getting removed.
created_at: 2025-04-16T10:50:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -78,6 +78,8 @@ def migrate_tables_with_logging(engine, diffs, table_names, adapters_by_table):
     migration_diffs = [diff for diff in diffs if diff.table_name in table_names]
     for table in table_names:
         adapter = adapters_by_table[table]
+        if not adapter.config.is_static:
+            adapter.config.save_build_not_required()
path: corehq/apps/userreports/pillow_utils.py
user: login: mkangia
body: :+1: 

Can this be done after the actual migration, in the for loop below?
And I believe this won't be needed anymore once the follow up PR is done, which will not set the flag only in case the data source does not need a rebuild.
created_at: 2025-04-16T10:57:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -753,6 +754,52 @@ def rebuild_failed(self):
     def rebuild_awaiting_or_in_progress(self):
         return self.meta.build.awaiting or (self.meta.build.is_rebuild_in_progress and not self.rebuild_failed)
 
+    def set_build_queued(self, *, reset_init_fin=True):
+        self.meta.build.awaiting = True
+        if reset_init_fin:
+            self.meta.build.initiated = None
+            self.meta.build.finished = False
path: corehq/apps/userreports/models.py
user: login: kaapstorm
body: Yeah, I didn't change the functionality.

I haven't tested this scenario yet, but I suspect that if there is a bug it wouldn't prevent rebuilds, it would just tell the user than an in-place rebuild is finished when it's actually still in progress. The other possibility is that we don't actually need to reset these values.

I don't think this stuff is covered by unit tests, and I'd like to fix that.
created_at: 2025-04-16T12:55:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -753,6 +754,52 @@ def rebuild_failed(self):
     def rebuild_awaiting_or_in_progress(self):
         return self.meta.build.awaiting or (self.meta.build.is_rebuild_in_progress and not self.rebuild_failed)
 
+    def set_build_queued(self, *, reset_init_fin=True):
+        self.meta.build.awaiting = True
+        if reset_init_fin:
+            self.meta.build.initiated = None
+            self.meta.build.finished = False
+
+    def save_build_started(self, *, in_place=False):
path: corehq/apps/userreports/models.py
user: login: kaapstorm
body: The asterisk means that `in_place` is a [keyword-only argument](https://docs.python.org/3.13/tutorial/controlflow.html#keyword-only-arguments). It indicates that you can't call this method with `self.save_build_started(True)`. You need to call it with `self.save_build_started(in_place=True)` so that it's clear what the boolean value means.

Here's what happens if you call a keyword-only argument as a positional argument:
```python
>>> def foo(bar, *, baz):
...     print(f'{bar}, {baz}')
... 
>>> foo('hello', baz='world')
hello, world
>>> foo('hello', 'world')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: foo() takes 1 positional argument but 2 were given
```

You also get [positional-only arguments](https://docs.python.org/3.13/tutorial/controlflow.html#positional-only-parameters), indicated with a forward-slash. e.g.
```python
>>> def foo(bar, /, baz):
...     print(f'{bar}, {baz}')
... 
>>> foo('hello', 'world')
hello, world
>>> foo(bar='hello', baz='world')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: foo() got some positional-only arguments passed as keyword arguments: 'bar'
```

You can find other examples in the codebase with
```bash
$ git grep -n ' \*, ' | grep '\.py'
```
created_at: 2025-04-16T13:12:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -753,6 +754,52 @@ def rebuild_failed(self):
     def rebuild_awaiting_or_in_progress(self):
         return self.meta.build.awaiting or (self.meta.build.is_rebuild_in_progress and not self.rebuild_failed)
 
+    def set_build_queued(self, *, reset_init_fin=True):
+        self.meta.build.awaiting = True
+        if reset_init_fin:
+            self.meta.build.initiated = None
+            self.meta.build.finished = False
+
+    def save_build_started(self, *, in_place=False):
path: corehq/apps/userreports/models.py
user: login: mkangia
body: Thanks,
I had forgotten about that :+1: 
created_at: 2025-04-16T13:34:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -260,7 +255,7 @@ def _iteratively_build_table(config, resume_helper=None, in_place=False, limit=-
 
 @task(serializer='pickle', queue=UCR_CELERY_QUEUE, ignore_result=True)
 def delete_data_source_task(domain, config_id):
-    from corehq.apps.userreports.views import delete_data_source_shared
+    from .views import delete_data_source_shared
path: corehq/apps/userreports/tasks.py
user: login: mkangia
body: I didn't know that isort was configured or itself prefers the dot notation for modules within the same app.
created_at: 2025-04-18T11:19:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1204,12 +1204,16 @@ def post(self, request, *args, **kwargs):
         try:
             if self.edit_form.is_valid():
                 config = self.edit_form.save(commit=True)
-                messages.success(request, _('Data source "{}" saved!').format(
+                message_list = [_('Data source "{}" saved.').format(
                     config.display_name
-                ))
-                messages.success(request, _(
-                    'This data source will be built / rebuilt automatically by CommCare HQ.'
-                ))
+                )]
+                if config.meta.build.awaiting:
+                    message_list.append(_(
+                        'This data source will be built / rebuilt automatically by CommCare HQ.'
+                    ))
+                else:
+                    message_list.append(_('This change does not require a rebuild.'))
path: corehq/apps/userreports/views.py
user: login: mkangia
body: nit: Can we either skip it or say that "HQ will not perform a data source rebuild for the changes made. Please initiate one manually if needed."

It appears that a rebuild might be needed at times, for instance, adding a nullable column is not considered to be rebuildable, so the user would need to rebuild separately if they want to populate existing rows
https://github.com/dimagi/commcare-hq/blob/0b590d2e093fe694e54d8a8ebdca0c79246c97c5/corehq/apps/userreports/alembic_diffs.py#L113
created_at: 2025-04-18T11:43:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -755,12 +758,35 @@ def rebuild_failed(self):
     def rebuild_awaiting_or_in_progress(self):
         return self.meta.build.awaiting or (self.meta.build.is_rebuild_in_progress and not self.rebuild_failed)
 
+    def set_rebuild_flags(self):
+        from .rebuild import get_table_diffs
+
+        if not self.rebuild_awaiting_or_in_progress:
path: corehq/apps/userreports/models.py
user: login: mkangia
body: The logic after this looks solid.
I am a bit unsure about this check here.

If there is a build awaiting, we don't need to set flags again, makes sense.
if there is a build in progress though, should not we set `awaiting` again?
Additionally, `rebuild_failed` is quite unreliable. I will be adding ticket to remove its usage soon.
created_at: 2025-04-18T11:49:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -755,12 +758,35 @@ def rebuild_failed(self):
     def rebuild_awaiting_or_in_progress(self):
         return self.meta.build.awaiting or (self.meta.build.is_rebuild_in_progress and not self.rebuild_failed)
 
+    def set_rebuild_flags(self):
+        from .rebuild import get_table_diffs
+
+        if not self.rebuild_awaiting_or_in_progress:
+            engine = connection_manager.get_engine(self.engine_id)
+            table_name = get_table_name(self.domain, self.table_id)
+            engine_metadata = get_metadata(self.engine_id)
+            diffs = get_table_diffs(engine, [table_name], engine_metadata)
+            tables_to_rebuild = {
+                diff.table_name
+                for diff in diffs
+                if diff.type in DiffTypes.TYPES_FOR_REBUILD
+            }
path: corehq/apps/userreports/models.py
user: login: mkangia
body: nit: I assume this is duplicated, so can it be extracted.
Fine to ignore if not duplicated.
created_at: 2025-04-18T11:50:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -755,12 +758,35 @@ def rebuild_failed(self):
     def rebuild_awaiting_or_in_progress(self):
         return self.meta.build.awaiting or (self.meta.build.is_rebuild_in_progress and not self.rebuild_failed)
 
+    def set_rebuild_flags(self):
+        from .rebuild import get_table_diffs
+
+        if not self.rebuild_awaiting_or_in_progress:
path: corehq/apps/userreports/models.py
user: login: mkangia
body: I see this was updated later to rely only on awaiting, so this is good.
created_at: 2025-04-18T11:55:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -755,12 +758,35 @@ def rebuild_failed(self):
     def rebuild_awaiting_or_in_progress(self):
         return self.meta.build.awaiting or (self.meta.build.is_rebuild_in_progress and not self.rebuild_failed)
 
+    def set_rebuild_flags(self):
+        from .rebuild import get_table_diffs
+
+        if not self.rebuild_awaiting_or_in_progress:
+            engine = connection_manager.get_engine(self.engine_id)
+            table_name = get_table_name(self.domain, self.table_id)
+            engine_metadata = get_metadata(self.engine_id)
+            diffs = get_table_diffs(engine, [table_name], engine_metadata)
+            tables_to_rebuild = {
+                diff.table_name
+                for diff in diffs
+                if diff.type in DiffTypes.TYPES_FOR_REBUILD
+            }
path: corehq/apps/userreports/models.py
user: login: kaapstorm
body: Yeah, there is a function that returns the same dictionary. 0f8db3b979e
created_at: 2025-04-18T12:28:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1204,12 +1204,16 @@ def post(self, request, *args, **kwargs):
         try:
             if self.edit_form.is_valid():
                 config = self.edit_form.save(commit=True)
-                messages.success(request, _('Data source "{}" saved!').format(
+                message_list = [_('Data source "{}" saved.').format(
                     config.display_name
-                ))
-                messages.success(request, _(
-                    'This data source will be built / rebuilt automatically by CommCare HQ.'
-                ))
+                )]
+                if config.meta.build.awaiting:
+                    message_list.append(_(
+                        'This data source will be built / rebuilt automatically by CommCare HQ.'
+                    ))
+                else:
+                    message_list.append(_('This change does not require a rebuild.'))
path: corehq/apps/userreports/views.py
user: login: kaapstorm
body: Good call. 7253e9aef29
created_at: 2025-04-18T12:28:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -260,7 +255,7 @@ def _iteratively_build_table(config, resume_helper=None, in_place=False, limit=-
 
 @task(serializer='pickle', queue=UCR_CELERY_QUEUE, ignore_result=True)
 def delete_data_source_task(domain, config_id):
-    from corehq.apps.userreports.views import delete_data_source_shared
+    from .views import delete_data_source_shared
path: corehq/apps/userreports/tasks.py
user: login: kaapstorm
body: Indeed. It's the ["LOCALFOLDERS" config here](https://github.com/dimagi/commcare-hq/blob/042f36b20451f0c340c5c0c1c45c4cefd0ecfcb0/.isort.cfg#L10). The purpose is so the local imports are grouped together below imports from other apps.
created_at: 2025-04-18T12:28:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -755,12 +758,35 @@ def rebuild_failed(self):
     def rebuild_awaiting_or_in_progress(self):
         return self.meta.build.awaiting or (self.meta.build.is_rebuild_in_progress and not self.rebuild_failed)
 
+    def set_rebuild_flags(self):
+        from .rebuild import get_table_diffs
+
+        if not self.rebuild_awaiting_or_in_progress:
+            engine = connection_manager.get_engine(self.engine_id)
+            table_name = get_table_name(self.domain, self.table_id)
+            engine_metadata = get_metadata(self.engine_id)
+            diffs = get_table_diffs(engine, [table_name], engine_metadata)
+            tables_to_rebuild = {
+                diff.table_name
+                for diff in diffs
+                if diff.type in DiffTypes.TYPES_FOR_REBUILD
+            }
path: corehq/apps/userreports/models.py
user: login: mkangia
body: okay, i thought the whole loop was same as in pillows.
created_at: 2025-04-18T15:24:04Z
author_association: CONTRIBUTOR

PR comments:
user: login: mkangia
created_at: 2025-04-15T09:08:41Z
body: thanks for the updates @kaapstorm 
The changes look sufficient to me. Is this ready for review?
author_association: CONTRIBUTOR
user: login: kaapstorm
created_at: 2025-04-15T09:33:13Z
body: > Is this ready for review?

Nearly. I'm going to rebase, and I'd like to do some testing. It should be ready later today.
author_association: CONTRIBUTOR
user: login: mkangia
created_at: 2025-04-15T10:27:40Z
body: If you are interested in exploring this a little further, I noticed that we could instead also choose to not set the flag if the data source won't be rebuilt anyway.
This would be done [here](https://github.com/dimagi/commcare-hq/blob/6552af589634e398d15f7c19a3db678ef2ad3742/corehq/apps/userreports/ui/forms.py#L292) by checking if the table will be rebuilt, similarly as done [here](https://github.com/dimagi/commcare-hq/blob/923e329ef9887d0192df0c3bac38a238ddbffbd4/corehq/apps/userreports/pillow_utils.py#L55).
This PR, the way it is currently, is still an improvement and this further improvement can be done as a follow up as well.
author_association: CONTRIBUTOR
user: login: kaapstorm
created_at: 2025-04-15T10:42:23Z
body: > we could instead also choose to not set the flag if the data source won't be rebuilt anyway.

Good idea!
author_association: CONTRIBUTOR
user: login: kaapstorm
created_at: 2025-04-17T19:10:07Z
body: I've reverted two previous commits, and fixed the way flags are set in commit https://github.com/dimagi/commcare-hq/pull/36186/commits/5db57238ea311ed7355c294f0b883d87547e35c2.

I added some tests in commit https://github.com/dimagi/commcare-hq/pull/36186/commits/34042dc46a6c2e91fa07a3559e3eceae0a7ca686.

The rest of the commits are just nits I found along the way. Still easier to review by commit.
author_association: CONTRIBUTOR
user: login: mkangia
created_at: 2025-04-18T15:25:45Z
body: oh wow, auto merge is dangerous.

Didn't realize the description was pretty outdated and it does not talk about how this has been tested.
Can you update it and share that this was tested?
author_association: CONTRIBUTOR