Querying AI and Cloud Trends: Azure and OpenAI Growth Slows  Amazon Growth Peaked in June
Source: tai_blog

Cutting through the AI hype to query actual developer usage (as new repos  so with presumptions) for prioritization of safety tools and partnerships.   TLDR (with caveats noted below):Public AI repos now appear as linear growth  not exponential (surge in March 2024 followed by rapid decline  now slower but steady).Azure/OpenAI public repo dominance: Azure shows 20x more new repos each month than the next leading hyperscaler  with OpenAI usage also dominating.Amazon Bedrock public repo growth may have peaked in June 2024 (slightly exponential until then).I leveraged GitHub repository creation data to analyze adoption trends in AI and cloud computing adoption. Code below  analysis follows.   Note on caveats:   Despite obvious bias and limitations (public packages and public repos containing only the names of these packages)  this method offers a unique view to developer adoption. Google Cloud and/or Microsoft formerly enabled querying of code within pages  which would have enabled a count of distinct import statements  but at some point recently this was disabled  therefore only leaving the repo names as queryable.   While imperfect  looking at repo creation provides enough data to challenge prevailing market narratives.   First  the notebook setup:Its only possible to use Google Cloud Platform (GCP) and BigQuery to access and query the GitHub data archive  so installed these packages (used colab initially  now parked in github).   # Install packages !pip install -q pandas seaborn matplotlib google-cloud-bigquery # Imports import pandas as pd import seaborn as sns import matplotlib.pyplot as plt from google.cloud import bigquery from google.oauth2 import service_accountQuery from GCP out of BigQuery:The following SQL extracts relevant data by categorizing repositories related to specific AI and cloud technologies  then aggregates repository creation counts by creation month.   Dependent on some manual investigation of the right python package names.   query =  WITH ai_repos AS ( SELECT repo.name AS repo_name  EXTRACT(DATE FROM created_at) AS creation_date  CASE WHEN LOWER(repo.name) LIKE '%bedrock%' THEN 'bedrock' WHEN LOWER(repo.name) LIKE '%vertex%' THEN 'vertex' WHEN LOWER(repo.name) LIKE '%openai%' THEN 'openai' WHEN LOWER(repo.name) LIKE '%anthropic%' THEN 'anthropic' WHEN LOWER(repo.name) LIKE '%langchain%' THEN 'langchain' WHEN LOWER(repo.name) LIKE '%azure%' THEN 'azure' WHEN LOWER(repo.name) LIKE '%llamaindex%' THEN 'llamaindex' WHEN LOWER(repo.name) LIKE '%neo4j%' THEN 'neo4j' WHEN LOWER(repo.name) LIKE '%pymongo%' THEN 'pymongo' WHEN LOWER(repo.name) LIKE '%elasticsearch%' THEN 'elasticsearch' WHEN LOWER(repo.name) LIKE '%boto3%' THEN 'boto3' WHEN LOWER(repo.name) LIKE '%ayx%' THEN 'ayx' WHEN LOWER(repo.name) LIKE '%snowflake-connector-python%' THEN 'snowflake' WHEN LOWER(repo.name) LIKE '%c3-toolset%' THEN 'c3ai' WHEN LOWER(repo.name) LIKE '%dataiku-api-client%' THEN 'dataiku' WHEN LOWER(repo.name) LIKE '%salesforce-einstein-vision-python%' THEN 'salesforce_einstein' WHEN LOWER(repo.name) LIKE '%qlik-py-tools%' THEN 'qlik' WHEN LOWER(repo.name) LIKE '%palantir-foundry-client%' THEN 'palantir_foundry' WHEN LOWER(repo.name) LIKE '%cuda-python%' THEN 'nvidia_cuda' WHEN LOWER(repo.name) LIKE '%openvino%' THEN 'intel_openvino' WHEN LOWER(repo.name) LIKE '%clarifai%' THEN 'clarifai' WHEN LOWER(repo.name) LIKE '%twilio%' THEN 'twilio' WHEN LOWER(repo.name) LIKE '%oracleai%' THEN 'oracle_ai' ELSE 'other' END AS keyword_category FROM `githubarchive.day.20*` WHERE _TABLE_SUFFIX >= '240101' AND _TABLE_SUFFIX NOT LIKE '%view%' AND type = 'CreateEvent' AND repo.name IS NOT NULL AND ( LOWER(repo.name) LIKE '%bedrock%' OR LOWER(repo.name) LIKE '%vertex%' OR LOWER(repo.name) LIKE '%openai%' OR LOWER(repo.name) LIKE '%anthropic%' OR LOWER(repo.name) LIKE '%langchain%' OR LOWER(repo.name) LIKE '%azure%' OR LOWER(repo.name) LIKE '%llamaindex%' OR LOWER(repo.name) LIKE '%neo4j%' OR LOWER(repo.name) LIKE '%pymongo%' OR LOWER(repo.name) LIKE '%elasticsearch%' OR LOWER(repo.name) LIKE '%boto3%' OR LOWER(repo.name) LIKE '%ayx%' OR LOWER(repo.name) LIKE '%snowflake-connector-python%' OR LOWER(repo.name) LIKE '%c3-toolset%' OR LOWER(repo.name) LIKE '%dataiku-api-client%' OR LOWER(repo.name) LIKE '%salesforce-einstein-vision-python%' OR LOWER(repo.name) LIKE '%qlik-py-tools%' OR LOWER(repo.name) LIKE '%palantir-foundry-client%' OR LOWER(repo.name) LIKE '%cuda-python%' OR LOWER(repo.name) LIKE '%openvino%' OR LOWER(repo.name) LIKE '%clarifai%' OR LOWER(repo.name) LIKE '%twilio%' OR LOWER(repo.name) LIKE '%oracleai%' ) ) SELECT FORMAT_DATE('%Y-%m'  creation_date) AS month  keyword_category  COUNT(DISTINCT repo_name) AS new_repo_count FROM ai_repos GROUP BY month  keyword_category ORDER BY month  keyword_category Then extract  load  transform  etc..Just created a pivot table with the right format..   # Query output to DF  create pivot df = client.query(query).to_dataframe() df['month'] = pd.to_datetime(df['month']) df_pivot = df.pivot(index='month'  columns='keyword_category'  values='new_repo_count') df_pivot.sort_index(inplace=True) # Remove the current month to preserve data trend by month df_pivot = df_pivot.iloc[:-1] Next  plotted the data:First time Id tried this  Id had to throw Azure to a secondary axis since it was 20x that of the next repo.   # Define color palette colors = sns.color_palette(husl  n_colors=len(df_pivot.columns)) # Create plot fig  ax1 = plt.subplots(figsize=(16  10)) ax2 = ax1.twinx() lines1 = [] labels1 = [] lines2 = [] labels2 = [] # Plot each keyword as a line  excluding 'azure' for separate axis for keyword  color in zip([col for col in df_pivot.columns if col != 'azure']  colors): line  = ax1.plot(df_pivot.index  df_pivot[keyword]  linewidth=2.5  color=color  label=keyword) lines1.append(line) labels1.append(keyword) # Plot 'azure' on the secondary axis if 'azure' in df_pivot.columns: line  = ax2.plot(df_pivot.index  df_pivot['azure']  linewidth=2.5  color='red'  label='azure') lines2.append(line) labels2.append('azure') # Customize the plot ax1.set_title(GitHub Repository Creation Trends by AI Keyword  fontsize=24  fontweight='bold'  pad=20) ax1.set_xlabel(Repo Creation Month  fontsize=18  labelpad=15) ax1.set_ylabel(New Repository Count (Non-Azure)  fontsize=18  labelpad=15) ax2.set_ylabel(New Repository Count (Azure)  fontsize=18  labelpad=15) # Format x-axis to show dates nicely ax1.xaxis.set_major_formatter(DateFormatter(%Y-%m)) plt.setp(ax1.xaxis.get_majorticklabels()  rotation=45  ha='right') # Adjust tick label font sizes ax1.tick_params(axis='both'  which='major'  labelsize=14) ax2.tick_params(axis='both'  which='major'  labelsize=14) # Adjust layout plt.tight_layout() # Create a single legend for both axes fig.legend(lines1 + lines2  labels1 + labels2  loc='center left'  bbox_to_anchor=(1.05  0.5)  fontsize=12) # Adjust subplot parameters to give specified padding plt.subplots_adjust(right=0.85)Results were interesting  since each month shows new repos created  Azure was exponential until March 2024  then declined quickly  is now linear growth since May 2024.   Re-plotted the data for clarity on smaller movements:With the top 3 repos removed  its easier to see the scale  Amazon Bedrock clearly shows steadier adoption but appears to peak in June 2024. Note that some packages are not meant to show adoption  since these are public packages (e.g. Snowflake  Nvidia CUDA)  and public repos.   # Isolate the top 3 to remove top_3 = df_pivot.mean().nlargest(3).index df_pivot_filtered = df_pivot.drop(columns=top_3) fig  ax = plt.subplots(figsize=(16  10)) for keyword  color in zip(df_pivot_filtered.columns  colors[:len(df_pivot_filtered.columns)]): ax.plot(df_pivot_filtered.index  df_pivot_filtered[keyword]  linewidth=2.5  color=color  label=keyword) ax.set_title(GitHub Repository Creation Trends by AI Keyword (Excluding Top 3 Packages)  fontsize=24  fontweight='bold'  pad=20) ax.set_xlabel(Repo Creation Month  fontsize=18  labelpad=15) ax.set_ylabel(New Repository Count  fontsize=18  labelpad=15) ax.xaxis.set_major_formatter(DateFormatter(%Y-%m)) plt.setp(ax.xaxis.get_majorticklabels()  rotation=45  ha='right') ax.tick_params(axis='both'  which='major'  labelsize=14) # Adjust layout plt.tight_layout() # Place legend outside the plot ax.legend(loc='center left'  bbox_to_anchor=(1.05  0.5)  fontsize=12) # Adjust subplot parameters to give specified padding plt.subplots_adjust(right=0.85) plt.show()Takeaways:   Very large disparity between the smaller packages and those from Big Tech.Azure and OpenAI dominate but growth is slowed.Amazon may have peaked in June 2024.More to come  stay tuned on more parts to this analysis (follow me for more updates)   FYI the dataframe is below  showing where obvious package names might not reflect the entire usage of the tool (e.g. Nvidia  Snowflake)  note (again) the many biases and caveats (one repo might contain x scripts etc)  so this assumes a new (and public) repo is growth.