Repository: dbt-labs/dbt-core

Issue:
[CT-2478] Excessive Template Rendering During Parse of Internal Analytics
Profiling `dbt parse` runs on our internal analytics project reveals repetitive rendering of simple string literals as templates. The problem is significant enough that patching it makes parsing a noticeable 12% faster. I'll open a PR shortly which mitigates this somewhat, but the list below should illuminate the nature of the problem.  Each string we are rendering as a template in native mode is included, prefixed by the number of times it is rendered. Each time that is done, the template is parsed, converted to python, and then executed. The word 'table' is going through this relatively slow process 41 times!

```
   2 ad_reporting
   2 amazon_ads
   2 amazon_ads_source
   2 analyses
   2 analysis
   1 analytics_dev
   2 apple_search_ads
   2 apple_search_ads_source
   1 assets
   1 audit_helper
   1 calogica/dbt_date
   2 calogica/dbt_expectations
   3 cloud_users
   1 coalesce
   1 codegen
   2 data
   1 datamocktool
   1 dbt
   2 dbt-labs/audit_helper
   1 dbt-labs/codegen
   1 dbt-labs/dbt_external_tables
   6 dbt-labs/dbt_utils
   1 dbt-labs/metrics
   1 dbt-labs/snowplow
  17 dbt-labs/spark_utils
   1 dbt_datamocktool
   1 dbt_date
   1 dbt_expectations
   1 dbt_external_tables
   4 dbt_modules
   8 dbt_packages
   1 dbt_pwebb
   1 dbt_snow_mask
   1 dbt_snowflake
   1 dbt_utils
   1 dbtlabs_pwebb
   1 delete+insert
   1 dev
   2 diff_when_downstream
   1 docs
   1 entechlog/dbt_snow_mask
   5 ephemeral
   1 externalbrowser
   2 facebook_ads
   2 facebook_ads_source
   2 facebook_pages
   1 facebook_pages_source
   1 fishtown_internal_analytics
   1 fivetran/ad_reporting
   1 fivetran/amazon_ads
   1 fivetran/amazon_ads_source
   1 fivetran/apple_search_ads
   1 fivetran/apple_search_ads_source
   1 fivetran/facebook_ads
   1 fivetran/facebook_ads_source
   1 fivetran/facebook_pages
   1 fivetran/facebook_pages_source
  18 fivetran/fivetran_utils
   1 fivetran/google_ads
   1 fivetran/google_ads_source
   1 fivetran/hubspot
   1 fivetran/hubspot_source
   1 fivetran/instagram_business
   1 fivetran/instagram_business_source
   1 fivetran/linkedin
   1 fivetran/linkedin_pages
   1 fivetran/linkedin_pages_source
   1 fivetran/linkedin_source
   1 fivetran/microsoft_ads
   1 fivetran/microsoft_ads_source
   1 fivetran/pendo
   1 fivetran/pendo_source
   1 fivetran/pinterest
   1 fivetran/pinterest_source
   1 fivetran/reddit_ads
   1 fivetran/reddit_ads_source
   1 fivetran/snapchat_ads
   1 fivetran/snapchat_ads_source
   1 fivetran/social_media_reporting
   1 fivetran/tiktok_ads
   1 fivetran/tiktok_ads_source
   1 fivetran/twitter_ads
   1 fivetran/twitter_ads_source
   1 fivetran/twitter_organic
   1 fivetran/twitter_organic_source
   1 fivetran/zendesk_source
   1 fivetran_utils
   1 fqn
   2 garage-snowflake
   2 google_ads
   2 google_ads_source
   2 hubspot
   1 hubspot_source
   1 incremental
   2 instagram_business
   1 instagram_business_source
   1 int_ad_reporting
   1 int_pendo
   1 integration_tests
   1 kw27752
   1 linkedin
   1 linkedin_ads
   1 linkedin_ads_source
   2 linkedin_pages
   1 linkedin_pages_source
   1 linkedin_source
   6 logs
  14 macros
   1 metrics
   2 microsoft_ads
   2 microsoft_ads_source
   1 mjirv/dbt_datamocktool
   4 models
   2 pendo
   1 pendo_source
   2 pinterest
   2 pinterest_source
   2 reddit_ads
   2 reddit_ads_source
   2 salesforce
   2 seeds
   3 selector
   1 single_tenant_freshness_test
   2 snapchat_ads
   2 snapchat_ads_source
   3 snapshots
   1 snowflake
   1 snowplow
   2 social_media_reporting
   1 source:bizzabo+
   1 source:slack+
   1 source:stitch_youtube+
   1 source:twitter+
   1 spark_utils
   1 stg_facebook_pages
   1 stg_hubspot
   1 stg_instagram_business
   1 stg_linkedin_pages
   1 stg_pendo
   1 stg_snowflake__users
   1 stg_tiktok_ads
   1 stg_twitter_organic
   1 sync_all_columns
  41 table
  20 target
   1 test
   4 tests
   2 tiktok_ads
   1 tiktok_ads_source
   1 transformer
   1 transforming
   2 twitter_ads
   2 twitter_ads_source
   2 twitter_organic
   1 twitter_organic_source
   1 user
  23 view
```



Discussion:

Resolving patch:
diff --git a/core/dbt/clients/jinja.py b/core/dbt/clients/jinja.py
--- a/core/dbt/clients/jinja.py
+++ b/core/dbt/clients/jinja.py
@@ -565,6 +565,8 @@ def _requote_result(raw_value: str, rendered: str) -> str:
 # is small enough that I've just chosen the more readable option.
 _HAS_RENDER_CHARS_PAT = re.compile(r"({[{%#]|[#}%]})")
 
+_render_cache: Dict[str, Any] = dict()
+
 
 def get_rendered(
     string: str,
@@ -572,15 +574,21 @@ def get_rendered(
     node=None,
     capture_macros: bool = False,
     native: bool = False,
-) -> str:
+) -> Any:
     # performance optimization: if there are no jinja control characters in the
     # string, we can just return the input. Fall back to jinja if the type is
     # not a string or if native rendering is enabled (so '1' -> 1, etc...)
     # If this is desirable in the native env as well, we could handle the
     # native=True case by passing the input string to ast.literal_eval, like
     # the native renderer does.
-    if not native and isinstance(string, str) and _HAS_RENDER_CHARS_PAT.search(string) is None:
-        return string
+    has_render_chars = not isinstance(string, str) or _HAS_RENDER_CHARS_PAT.search(string)
+
+    if not has_render_chars:
+        if not native:
+            return string
+        elif string in _render_cache:
+            return _render_cache[string]
+
     template = get_template(
         string,
         ctx,
@@ -588,7 +596,13 @@ def get_rendered(
         capture_macros=capture_macros,
         native=native,
     )
-    return render_template(template, ctx, node)
+
+    rendered = render_template(template, ctx, node)
+
+    if not has_render_chars and native:
+        _render_cache[string] = rendered
+
+    return rendered
 
 
 def undefined_error(msg) -> NoReturn:


Test patch: