CATransformers: Carbon Aware Transformers Through Joint Model-Hardware Optimization
Authors: Irene Wang
Mostafa Elhoushi
H Ekin Sumbul
Samuel Hsia
Daniel Jiang
Newsha Ardalani
Divya Mahajan
Carole-Jean Wu
Bilge Acun
Venue: NeurIPS 2025 poster ()
Keywords: Carbon Footprint
Sustainability
Hardware Accelerator
Architecture Search
Neural Architecture Search
Optimization
Primary area: optimization
TL;DR: We propose CATransformer, the first carbon-aware co-optimization framework for Transformer-based models and hardware accelerators.

Abstract: Machine learning solutions are rapidly adopted to enable a variety of key use cases, from conversational AI assistants to scientific discovery. As the adoption of machine learning models becomes increasingly prevalent, the associated lifecycle carbon footprint is expected to increase, including both *operational carbon* from training and inference and *embodied carbon* from AI hardware manufacturing. We introduce CATransformers, the first carbon-aware co-optimization framework for Transformer-based models and hardware accelerators. By integrating both operational and embodied carbon into early-stage design space exploration, CATransformers enables sustainability-driven model architecture and hardware accelerator co-design that reveals fundamentally different trade-offs than latency- or energy-centric approaches. Evaluated across a range of Transformer models, CATransformers consistently demonstrates the potential to reduce total carbon emissions --by up to 30\% -- while maintaining accuracy and latency. We further highlight its extensibility through a focused case study on multi-modal models. Our results emphasize the need for holistic optimization methods that prioritize carbon efficiency without compromising model capability and execution time performance. The source code of CATransformers is available at https://github.com/facebookresearch/CATransformers.

CATransformers: Carbon Aware Transformers Through Joint Model-Hardware Optimization
Irene Wangl.24 Mostafa Elhoushi? H. Ekin Sumbul Samuel Hsia", Daniel Jiang', Newsha Ardalani Divya Mahajan', Carole-Jean Wu?, Bilge Acun?
Georgia Institute of Technology, FAIR at Meta, RReality Labs Meta, *Meta, "Work done at Meta irene wang@gatech. edu acun@meta.c com
Abstract
Machine learning solutions are rapidly adopted to enable variety of key use cases, from conversational assistants to scientific discovery. This growing adoption is expected to increase the associated lifecycle carbon footprint, including both operational carbon from training and inference and embodied carbon from AI hardware manufacturing. We introduce CATransformers the first carbon- aware co-optimization framework for Transformer based models and hardware accelerators. By integrating both operational and embodied carbon into early-stage design space exploration, CATransformers enables sustainabillty-driven model architecture and hardware accelerator co- co-design that reveals fundamentally different trade-offss than latency- or energy-centric approaches. Evaluated across a range of Transformer models, CATransformers consistently demonstrates the potential to reduce total carbon emissions-by by up to 30%-while maintaining accuracy and latency. We further highlight its extensibility through focused case study on multi-modal models. Our results emphasize the need for holistic optimization methods that prioritize carbon efficiency without compromising model capability and execution time performance. The source code of CATransformers is available at htt.:p//gggtthh..ce//eeeeeeeeere
Introduction
As machine learning (ML) systems become more widespread across various industries, it is crucial to take a closer examination of their carbon footprint and find strategies to mitigate across the system stack. Sustainable ML system design requires holistic approach that considers both operational carbon (energy used during training and inference) and embodied carbon (emissions from hardware manufacturing and lifecycle) |GEH 22). which together contribute to total emissions. This work tackles the question: How does incorporating carbon footprins metrics into optimization workflows influence the design of ML models and hardware architectures a co-optimization?
fundamental challenge designing sustainable Al systems stem from the tight coupling between model architecture and hardware design, which together shape both operational and embodied carbon. Model execution on a particular hardware determines runtime characteristics, such as compute intensity and memory access patterns, which affect operational carbon, while hardware parameters like chip area, memory hierarchy, and fabrication technology drive embodied carbon. Carbon optimization requires joint model-hardware exploration, particularly during carly accelerator design, to uncover opportunitics that align model demands with hardware capabilities in a carbon-effieient more important for multi-modal workloads like vasion-language models. Each modality introduces distinct bottlenecks-e.g. vision transformers usually have compute-boune bound workloads (DGC24) while text transformers have more memory- -boud workloads [Fu24] Co-
way. Co-design
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

60
50
CarbonCLIP-S
TinyCLIP-39M
Accuracy (%)
CarbonCLIP-XS
40
Same accuracy: 17% less carbon <15ms lafency
8% better accuracy: less carbon latency
30
TinyCLIP-SM
20
0.30
0.35
0.40
0.45
Carbon (kg CO2e)
Figure CarbonCLIP models achieve lower carbon footprint and higher accuracy compared to baseline CLIP models.
optimization across modalities and system layers is essential to minimize the total carbon footprint without compromising performance.
While this sustainability challenge applies across the entire Al lifecycle, the rapid adoption of AI on edge devices presents a particularly acute and fast-growing problem (QPFM F 23 22L
With billions of products like smartphones and AR/VR headsets requiring specialized, custom accelerators [WLS SWL+22 [AI25 Goo20 Nvi25], the embodied carbon from manufacturing constitutes a massive, upfront environmental cost. Furthermore, the cumulative operational carbon from continuous inference over a device's ' multi-year lifespan represents a substantial and often overlooked energy demand (App2I). Addressing the tightly coupled model-hardwand system at the edge is therefore critical first step, offering a high-impact domain develop and validate the carbon-aware co-design principles needed for sustainable Al.
Prior hardware-aware neural architecture search (NAS) methods have focused primarily latency or energy and are not suitable for total carbon optimization. This is because carbon is complex metric that cannot be captured by optimizing latency or energy isolation. These approaches either neglect embodied carbon by optimizing only the model on fixed hardware [WWL*20 or treat latency, energy, and chip area as orthogonal constraints (ZDM 22), overlooking the tightly coupled nature of operational (latency- and -drvven) and embodied (chip area-driven) carbon. As result, these methods are fundamentally limited in their ability to minimize total carbon emissions. Furthermore, prior work often assumes fixed hardware or fixed model backbones, limiting their applicability to broader range emerging hardware platforms and modern, multi-modal models.
We introduce CATransformers--llee first framework to jointly optimize model and hardware archi- tecture with the specific goal of minimizing total carbon emissions for edge inference-only devices. Distinct from prior latency- or energy-centri methods, our approach incorporates both operational and embodied carbon into a unified optimization pipeline, revealing carbon-efficient configurations previously overlooked. We observe that optimizing traditional metrics in isolation frequently results suboptimal trade-offs. For example, accelerating inference by increasing compute density may inadvertently increase power consumption and embodied carbon due to larger chip area or more resource-intee inttnnsve fabrication. By prioritizing total carbon as first-class objective, CATransformers yields fundamentally different and more sustainable design choices.
CATransformers systematically explores the joint design space of model architectures and hardware accelerators using multi-objective optimization. It comprises three main components: (1) a Multi- Objective Bayesian Optimizer that balances accuracy, latency, energy, and carbon emissions; (2) an ML Model Evaluator that efficiently navigates model variants using importance-basse pruning and fine-tuning; and (3) a Hardware Estimator that profiles latency, energy, and carbon footprint.
We evaluate CATransformers on language, vision, and multi-modal Transformer-bant -base models and configurations reduce total carbon by 30% over latency-
show that its co-optimized optimized and 8% over energy-optimin baselines. To further highlight its utility, we present CarbonCLIP, family of CATrassfornes-optiiizzd CLIP models.
Figure D shows CarbonCLIPP achieves up to 17% lower total carbon emissions compared to state- of-the-art CLIP variants on edge devices, while maintaining similar accuracy and latency. Joint

---

optimization enables low-carbon, high high-accuracy designs that outperform approaches that optimize only hardware configurations.
Our key contributions include:
1. Insights and Analysis:
Empirical insights showing how carbon- aware optimization shifts model-hardware trade-offs relative to traditional metrics, enabling early-stage design explo- ration for next-generation ML accelerators.
2. Quantification Framework:
We develop the first open-source toolchain to estimate both operational and embodied carbon for custom accelerators during design-space exploration.
3. Carbon-Aware Co-optimization: CATransformers uses multi-objective Bayesian opti- mization to jointly explore model and hardware design spaces, balancing accuracy, latency, energy, and carbon. It leverages fine-tuning based proxy signals to reduce evaluation cost.
4. Sustainable Multi-Modal Models:
Using CATransformers, we co-optimize CLIP variants (CarbonCLIP) that reduce total carbon by up to 17% compared to edge-deployed CLIP baselines, without sacrificing accuracy or latency.
CATransformers is designed as modular, extensible framework that supports the seamless integra- tion of diverse model architectures, optimization strategies, and carbon estimation techniques. The goal is not outperform other NAS frameworks on traditional metrics, but highlight the fundamen- tally different design trade-off's that emerge under carbon-centric optimization objectives. Through systematic comparisons across multiple optimization modes and model families, we demonstrate the practical impact of carbon-aware design, which is the core contribution of this work.
2 Background and Related Works
Hardware Accelerator Search: Specialized hardware accelerators have been developed to efficiently
deep learning workloads, with tensor cores for matrix operations [J MPA 16 PSM 17
CKES16) and vector cores for element-wise operations [JYP+ GKX 24). Prior accelerator search
frameworks [APK 24| |MK 22 WTPM24 PMT25 ZHS 22, SSL23 optimize for throughput
and energy, but do not jointly optimize hardware and model architectures or consider carbon footprint in the design process.
Hardware and Neural Architecture Co-optimization: Co-optimization methods [ZDM 22
CHY+21 21 + LYH2I] typically focus latency energy, ignoring carbon as primary design objective. Approaches that do consider carbon [EAA 233 ECA #25 ZG23] often optimize either the model hardware in isolation. Furthermore, prior work often assumes fixed hardware or fixed model backbones, limiting applicability to broader range of emerging hardware platforms and modern, heterogeneous models partill-rrticularrly multi-modal architectures. Our work addresses this gap by jointly optimizing both hardware and model architectures with total carbon foolprint-including including embodied and operational emissions-as a first-class objective.
The Carbon Footprint of AI Systems: Much of the research on the carbon footprint of AI fo- cuses operational emissions, with three main directions: (1) quantifying the emissions from WFRG*22), (2) analyzing energy use during deployment and inference S24 LGH 24 and (3) developing techniques to reduce emissions while exploring
training [BLL23 SGM20 LLSD19 LLSD19
energy-performance trade-offs GHI 25 ZLJG24 IJGT24 ALK
Embodied carbon, however, remains underexplored. Tools like ACT 22H+22) IMEC.netzero [MM25]] and LLMCarbon (FKW 24) have begun to estimate embodied emissions, but a holistic framework that quantifies and minimizes both operational and embodied carbon remains lacking. CORDOBA ECA 225 introduces a carbon-aware accelerator design tool, but is not open-sourced, limiting reproducibility and extensibility. Unlike CE-NAS [ZL4 which reduces carbon emissions during neural architecture search, CATransformers is the first to optimize total carbon emissions from both model and hardware, during etrly-stage design.
AI Systems on the Edge: AI models are no longer confined to large-scale datacenters and are now widely deployed edge devices like smartphones, AR/VR headsets, robotics, and vehi- cles [3 S21 [A[25]. These edge settings present unique challenges, including strict resource constraints, hardware heterogeneity, and communication bottlenecks that impact
3

---

Output
Base Model
Multi- Mutti-Objective Bayesian Optimizer
Architecture Template
Submodel Config
Proxy Accuracy
Submodel HW Config
Latenoy, Energy. Carbon
Model Config
Metrics and Constraints
ML Model Evaluator
HW Estimator
IITIT
Hardware Config
Figure 2: Overview of the CATransformers framework. The Bayesian optimizer iteratively explores model and hardware configurations using accuracy, carbon, and latency estimates from evaluation modules, outputting optimized co-designs.
performance [WNM23] HYW bJRE 23). This wide adoption, however, also raises significant environmental concerns. The sheer scale of device production means embodied carbon from chip fabrication is major contributor to total emissions [KK, Simultaneously, the cumulative operational carbon from frequent inference over device's multi-year lifespan presents a distinct and growing sustainability challenge. CATransformers aims to address this gap by enabling holistic, carbon-aware co-design of AI models and hardware accelerators specifically for edge systems.
CLIP Models and Edge Variants: Multi-modal models like CLIP RKH combine Transformer- based text encoders with image encoders (ResNet [HZRS16] or ViT [DBK 21)) and are trained on large datasets (SVB 21 XXT 24 SBV+22 GIF 23 learn cross moodll associations. These models support tasks like zero-shot classification and retrieval. Several variants [WPZ*2) LBL* STJ 23 VPF 24] adapt CLIP for edge devices through pruning and efficient training. but focus solely on accuracy and latency. Our work is the first to optimize CI.IP models for edge deployment with total carbon emissions in mind, without sacrificing accuracy or latency.
3 Framework Overview
In this section, we introduce CATransformerss a curbon-aware architecture search framework for sustainability-drieee co-optimization of ML models and hardware architectures. As illustrated in Figure 2 CATransformers framework takes three inputs: (1) a base ML model, (2) a hardware architecture template, and (3) set of optimization objectives and constraints that define the joint modee-hardware search space. The framework consists of three key components: a multi-objective optimizer, MI model evaluator, and a hardware estimator.
3.1 CATransformers Inputs
Base Model:
The base model is large, pre-trained Transformer that serves the foundation for generating pruned variants. It defines the architecture's 's shape, functionality, and search space. As shown in Figure 3a pruning is applied along several dimensions, including the number of layers, hidden size, feedforward network width, attention heads, and embedding dimensions.
Hardware Architecture Template: The accelerator template (Figure 3b) captures key components and tunable parameters inspired by academic and industry designs JYP 17
APK 72S WTPM24). It features tensor cores with Processing Elements (PEs) arranged in X and Y dimensions for GEMM operations, vector units for element-wise operations, local buffers for data reuse, and shared global SRAM and off-chip memory consistent with existing edge inference accelera- tors [SWL*22 WLS$$24 Increasing PEs or cores boosts performance but raises area and energy costs; more on-chip memory lowers latency but adds energy overhead. The optimal hardware config- uration depends on the model architecture, size, and performance constraints. Table summarizes the hardware design space parameters.
3.2
MT. Model Evaluator
The ML model evaluator estimates the accuracy of candidate model architectures during the search.

---

Layer
Core
Core
Layer
Layer
FTN
Core
Core
FFN
Vider
Memory
Ruber
Embeddiing
Off-chip DRAM
(a) Overview of the pruning dimensions
(b) Architecture template
Figure Overview of model pruning dimensions and hardware template for CATransformers
Table 1; Architecture design space parameters.
Parameter Description
Notation
Design Space (S)
Number
TC
powers ol
PE. Array dim
PE,
to 256 powers
PE Arry dim
PE,
powers 2
Buffer
powers
Local Buffer Size
1.2
256 KD to MB powers of'2
Bandwidth
L2ww
words/eycle
Vector Unit width
Vp
PE,
Fixed Parameters
Global Bandwidih
GLW
156 wordscycle
Off-chip DRAM Size
HBM
GB
Technology
BitWidih
I
Maximom TOPS
Timx
Frequency
500 MI
Pruning:
The evaluator prunes the pre-trained base model along key dimensions, including the number of layers, feedforward network size, attention heads, and embedding dimension. For multi- modal models like CLIP, each Transformer (text and vision) pruned independently using strategies from prior work [LBI_ 24 SDDN20 WWL++20) Each layer is pruned uniformly. See Appendix A
for pruning details and empirical observations.
Direct evaluation of untrained pruned models results in poor accuracy, while fully retraining every model is computationally infeasible. Instead, we use lightweight fine-tuning to approximate accuracy. Our ablation studies show that this method yields a high Spearman correlation (0.98) with fully trained accuracy, making it reliable proxy for ranking candidate models (Appendix{) C). This approach allows accurate and efficient evaluation within the optimization loop, enabling scalable exploration of Transformer-based models without full retraining. For language models we use the MRPC task from the GLUE dataset [WSM+ 18) for semantic similarity detection, for vision models, the CIFAR-10 dataset [Kri09] for image classification; and for CLIP models, the MSCOCO dataset [LMB+ 15) for retrieval.
Fine-tuning for Accuracy Proxy:
3.3 Hardware Estimator
The hardware estimator provides unified, end-to-end analysis of inference latency and total carbon footprint-includdin both embodied and operational emissions-for each model-hardware configu- ration. It leverages performance and energy models to estimate operator-level latency, energy, and area [WES19 ONFL23), and incorporates carbon modeling for manufacturing-relitt emissions and location-specde -specffc operational carbon, scaled over the system's 's deployment lifetime [GHH*22 Map25]. To the best of our knowledge, this represents the first open-source toolchain built for early-stage design space exploration that holistically quantifies latency, energy, and total carbon emissions of custom MI. accelerators. detailed workflow provided in Appendix
3.4 Multi-Objective Optimization
CATransformers uses multi-objective Bayesian optimization to efficiently explore the joint de- sign space of model architectures and hardware configurations building Ax [MP24] and
5

---

BoTorrh [BKJ 20] with the qNEHVI algorithm [DB22]] Compared to reinforcement learning or evolutionary search, Bayesian optimization offers better sample efficiency and uncertainty modeling, making it ideal for our static, high-dimensional search space where each configuration takes 5-15 minutes to evaluate. While prior work [WWL. =0 explored just 125 models over 30 iterations, CATraasformers scales to a search space of 100 million configurations in only 100 iterations.
The optimization maximizes accuracy while minimizing latency, energy, and total carbon (embodied + operational), exploring model (Figure 3a) and hardware parameters (Table I under a rOPS-based compute budget based on publicly available edge accelerators. Our framework supports four compute- constrained modes: (1) Accuracy & Total Carbon (with latency constraint), (2) Accuracy & Latency, (3) Accuracy & Energy, and (4) Accuracy, Latency, & Total Carbon.
3.5 Outputs
CATransformers outputs optimized combinations of model and hardware configurations that, when deployed together, improve overall efficiency. Once the optimizer identifies carbon-efficient, pruned model architectures, these models can optionally be fine-tuned to recover any accuracy loss. Pre- liminary results show these pruned models require far fewer training steps than full pre-training. Specifically, CarbonCLIP models are fine-tuned for just 2 epochs on MetaCLIP [XT+ 24), using only 40% of the training steps compared to prior works 21 XXT 24
WPZ+
4 Evaluation
4.1
Experimental Settings
To demonstrate the versatility of CATransformers, apply it diverse transformer- based architectures: encoder-only Bert-Base [DCLT19], decoder-only Llama3-8B [GDJ+24], vision transformer V [DBK+21), and multi-modal CLIP models (CLIP-ViT-B/16 and CLI--ViT- (K We use pre-trained weights from HuggingFace [WDS+ for Bert, Llama, and ViT. and adopt OpenCLIP's [IW DataComp- IB [GIF+23) models for CLIP. Building on CLIP optimization results, we generate CarbonCLIP models with corresponding hardware designs.
Model:
Prior work primarily targets performance or energy efficiency, rarely considering carbon- aware hardware-mode co-design. These approaches are typically evaluated on fixed hardware and model architectures, making direct comparison difficult. To ensure fair and meaningful evaluation, we benchmark CarbonCLIP against two widely adopted baselines: (1) standard CLIP models (ViT- B/16, ViT-B/32), and (2) TinyCLIP, state-of-the-art variant optimized for latency. For each, we apply CATransformers's 's hardware search with fixed model architectures to derive comparable accelerator designs. This allows to isolate the benefits of carbon-aware co-optimization. Our results show that carbon-optmmized models can match or exceed the accuracy and latency of traditional Daselines-dmmonstrraie the practical value of our approach.
Baselines:
Hardware Estimation and Validation: We estimate accelerator area, latency, and energy using open-source tools (Appendix [B). assuming 22nm process technology and 8-bit integer operations. To validate latency estimates, we use SCALE-Sin [SZW*18 SJZ for CLIP's QKV projections. Our estimates have average latency error of 13%, consistent with prior work (PRS 19 WES19), and are validated on standard compute array sizes (16x16 to 64x64). We further validate energy and latency predictions against real GPU hardware (V100, A A100, H100) by modeling GPU-like architectures. Our estimates closely match measured results, with average errors of 8% (energy) and 9% (latency), and Spearman's 's rank-order correlation r ranging from 0.5 to 1.0 and p < 0.05, confirming toolchain accuracy. Details of the validation is in Appendix M
Operational carbon is estimated using California grid intensity over 3-year lifespan (1 inference/sec, 6 hrs/day), reflecting typical mobile usage [App21] NSY24 Har25) Embodied carbon is caleulated assuming fabrication in Taiwan. Appendix explores the impact of different deployment regions, energy sources, and optimization metrics.
Carbon Emission Estimation:
Execution Setup: Bayesian optimization is performed on single node (8xV100 GPUs, 80 CPUs) over 100 trials, taking 5-20 hours depending on the model. Post-pruning training of CarbonCLIIP uses 224 GPUs the MctaCLIP-2.5B dataset XXT+ 24), with batch size of 128, learning rate of

---

Carbon
Carbon Latency
Energy
Latency
Lateney (ma)
CLIP-
BERT
Llama3
VIT-B16
VIT-B16
20
20
0.48
0.50
0.30
0.32
0.34
0.00
0.33
0.25
0.38
0.40
0.33
0.55
040
0.0
Carbon (kgCO2e)
Figure 4: ISO-Accuracy plot showing the latency-carbon trade-off across optimization strategies, with accuracy matched within +1%.
5 x 10-1, and distillation epochs. We quantify the Carbon footprint of running CATransformers in Appendix N
4.2 Joint Model and Hardware Architecture Search Using Different Metrics
In this section, we use CATransformers to perform joint model-hardware architecture search, with carbon footprint as a central design metric alongside traditional objectives like accuracy, latency, and energy. We evaluate different optimization modes under a 20 TOPS compute budget, representative of modern edge accelerators [HA125] Nvi25). When latency is not optimization target, maximum latency constraint of 50ms is enforced [vMV]22] to ensure realistic specifications. Each experiment repeated three times, and we compute the Hypervolume (HV) indicator to assess consistency. HV measures the portion of the objective space dominated by the Pareto front relative to a reference point. Across runs, its standard deviation is below 0.03, with an average coefficient of variation under 3.5%, indicating statistically consistent results. Appendix F) a detailed breakdown of the statistics.
Takeaway I: Carbon optimization yields the lowest footprint but at the cost of latency; energy optimization strikes a better balance.
Figure 4 shows the latency-carbon trade-off at iso-accuracy points (based on proxy accuracy before fine-tuning). Full configurations across additional accuracy levels for each model are detailed in Appendix D Comparing optimization modes, we find that carbon optimization significantly reduces carbon footprint by an average of 30%, but with a 7.7x increase latency compared to latency- optimized baselines. Energy-optimizer configurations reduce carbon by 24% while limiting latency overhead 4x. Joint optimization for Carbon and Latency achieves 18% carbon reduction with minimal latency increase, effectively balancing sustainability and performance.
Takeaway 2: Energy optimization indirectly reduces latency, but as effective as direct latency opfimization.
Minimizing energy tends to reduce both power and computation time, often lowering latency. In contrast, carbon-focused optimization emphasizes minimizing hardware area to reduce embodied carbon, often resulting in slower, smaller designs. Because total energy consumption is proportional to latency (i.e., energy = power x delay), latency cannot be extended arbitrarily when using smaller-area hardware. Excessively long latencies not only fail to meet practical performance targets but also begin to significantly increase operational carbon costs. Still, energy optimization does not always guarantee low latency. For example, Bert and Llama3 configurations in Figure show higher latency despite energy-focused design. Direct latency optimization, seen in Latency-only and Carbon+L LLatency modes, consistently delivers the lowest-latency results across all models. These trade-offs highlight the need to carefully balance carbon constraints and real-world performance demands.
Takeaway 3: Latency-optimized designs favor large, high-throughput hardware; carbon-optimized designs use compact, low-power accelerators.
cueency-focused designs often use up to 3x larger area, enabled by more compute units and larger memory hierarchies. These systems frequently select larger models that fully utilize hardware to meet latency targets. In contrast, energy optrgieop designs typically pair smaller models with larger accelerators minimize delay and energy per inference. Carbon-optimized designs prioritize smaller hardware to reduce embodied carbon, at the cost of performance. For example, on BERT- Base (Figure 4) the carbon-optimized system uses a single-core accelerator with 512 Processing Flements (PEs), 64KB local memory, and 2MB global memory. The latency-opiimized design uses two cores,
7

---

Table 2: Hardware architecture search for fixed baselines CLIP model architectures.
Minimmi Carhon
Latescy
Model Architceture
Params (M)
Carbon (ggC)2a
Litexy (ms)
Handware Architecture
Carbon (gC022e
Latescy (ms)
Hardwaee Archioeowr
Coress
Dimension
Cire
Meneey Conlin
[Local, Global)
Cores
Dimension
Cire
Menwey Ceeliy
[Local, Global)
CLIP-B/16
149
0.34
18.3
[256,3]
64 KB. 2hBB
1,00
3.4
2
(256,10)
258KB, AMB
CLIP
427
1.43
68.7
2
(128.16)
128 KB. 2MB
1.75
66.4
(68.64)
256KB. 4MB
CLIP-H/14
(128,32)
128KB, 4MB
2,60
70.2
512KB, 4MB
T
0.34
(256.4)
0.56
1.3
(256/64)
256KR,
1 39M816
83
0.64
944
(256,8)
64KB, 2MB
0.59
2.2
(256.16)
125KD. 4MB
ThyyCLIP
(32,52)
64KB,
64KB, 2MB
6
(128.8)
0.49
1.4
(128.32)
6:KB, 2MB
each with 4K PEs, 128KB local memory, and 4MB global memory-cutinng latency by over 4x but increasing carbon footprint by 26%.
Takeaway 4: Model sensitivity to pruning varies by model architecture: CLIP and ViT are especially sensitive hidden dimension pruming.
Each model architecture responds differently to pruning. CLIP and ViT models are particularly sensitive to reductions in the hidden dimension, leading CATransformers to typically follow this pruning order for these models: FFN dimension -+ attention heads -> layers -+ embedding dimen- sion. This sensitivity arises because VIT's split images into relatively few patch tokens, requiring each token to earry rich semantic and spatial information. The hidden dimension governs the ex- pressiveness of these token embeddings, and pruning it severely limits ViT's ability to model visual content-especially since ViTs lack the hierarchical features of CNNs and rely on global attention.
CLIP models are especially sensitive due to their multi-modal nature, where precise alignment between visual and textual embeddings crucial. Pruning hidden or embedding dimensions can sigmificantly degrade performance (Appendix[A) ( as excessive pruning of the vision encoder disrupts alignment with the text encoder, harming model accuracy. Additionally, the text encoder often pruned more aggressively than the vision encoder, which is essential for processing complex visual inputs. contrast, language models such as Bert and Llama3 tend to be more robust to pruning in the hidden dimension. These observations underscore the challenges of pruning multi-modal models like CLIP and motivate our focus on their optimization throughout the paper.
4.3 Hardware Optimization for Fixed CLIP Models
While the previous section highlighted CATransformers' general utility, we now focus on CLIP-a widely used multi-modal model combining text and vision encoders. CLIP poses additional challenges due to its heterogeneous computation and the complex interplay between modality-specific pruning and system performance. We first evaluate the carbon footprint of state-of-the-ar art CLIP variants. To ensure fair comparison, we use CATransformers' hardware search to optimize inference latency and carbon footprint under fixed model and 20 TOPS compute budget. This yields a Pareto frontier illustrating optimal diteccy-crrbo trade-offs. Table summarizes configurations with minimum carbon and minimum latency for each model. As with joint optimization, design results in smaller accelerators with fewer compute and memory resources. While this reduces area and emissions, it typically increases inference latency.
hardware
Takeaway Hardware must be tailored to the model-parumeter count, patch size, and modality- specific traits affect optimal configurations.
Results from the exedd-moel hardware search underscore the importance of hardware-mode co- design: optimal performance and carbon efficiency require aligning hardware with model architecture. ViT patch size notably influences accelerator dimensions. For instance, TinyCLIP-6 111PP6111322-es11/111 having more parameters outperforns paired with hardware better suited its larger patch size. ViT-B/32 produces a sequence length of 50, compared to 197 for ViT-B/16, enabling more efficient execution on smaller accelerators. Embedding dimensions and patch sizes shape execution patterns, making some models inherently better matched to specific hardware. Memory configuration also scales with model size: smaller models perform well with 64KB local and 2MB global buffers, while larger models like CLIP-L/114 and CLIP-H/144 require at least 1 128KB local buffers and larger global memory.
WWVCLIP-39M//1 in both latency and carbon footprint when
8

---

Table 3: Hardware and model architecture properties of each variant of the CarbonCLIP family. Hardware configurations are specified as: {TC, PE,, yE L.2, L2bw. GLB). PE denotes Processing Element. Text and Vision encoders specified as: {Num Layers, FFN Dim, Hidden Dim, Num Heads);
Name
Carbon (gCCO2c)
Latency (ms)
Hardware Configuration
Model Conliguration
Avg. Accuracy
Text Encoder Conliguration
Vision Encoder
over 41 datasets
Configuration
CLIP-R/16 DaraComp
0.54
18.5
[1. 256, N, 128,
(12, 2048, 512, 8]
3072, 768,
0.34
3,0
[2. 256, 4, 64, 32, 1
[3, 1024, 256, 4)
[10, 1024, 256, 4]
41
30.7
ThnyCI 3P3369
0.44
9.4
128,
2048, 512,
SarbonnTLIPPXX
0.32
7.1
[1. 256, 4,
21
[6. 1024, 284, 4)
[6. 1536, 576, 6]
41
38.7
C1S
0.36
12.0
25,
4,
64
1024,
1920, 672, 6]
CarbonCT.IP-M
0.39
19.7
[1, 256,
4, 64,
128.2
[8, 1536, $12, 6)
2304, 672, 6]
79
47.9
CarbomC1.1P.1
0.42
13.7
256,
128,
1280,
768.
48.7
CLP
0.49
19.1
[1, 256,
8, 64, 128, 21
(12, 2048, 512, 4]
[12, 3072, 768. 5]
123
52.0
0.5
0.5
car carton 100ms
0.4
0.4
0.4
lutorcy
caroon blercy
Kerunnyy (h6)
enargy
0.3
0.3
Accrsccy
0.3
20 TOPS
4 TOPS
0.2
02
carbo- 100mts
litenyy
latency
ussncy
1 TOPS
40d
0.5
22
033
0.4
5,5
0.0
0.4
0.5
0.6
67
Carton Carbon (kg002e)
Co too CogC
Cartion Rgg002e)
Figure 5: Pareto frontiers under varying compute constraints.
4.4 Accuracy Evaluation
We compare CarbonCLIP models to TinyCLIP and the CLIP-VTT-B/16 baseline (pretrained on DataComp-1B), using each baseline's most carbon-efficient configuration. TableB the Car- bonCLIP family, selected from the Pareto frontiers in Appendix E For each CarbonCLIP model, We perform post-pruning training, and evaluate their performance on 41 zero-shot tasks from the CLIPBenchmark [LA22]. The table summarizes model and hardware configurations for CarbonCLIP. XL to CarbonCLIP XS (largest to smallest models), along with their carbon footprint, latency, and average accuracy. Full per-dataset results are in Appendix H We also extend the CarbonCLIP family the CLIP-B/32 architecture for comparison with EmyCLiP-B//3 baselines (Appendix G
CarbonCIIB models outperform baselines across a range of carbon budgets. Notably, CarbonCLIP-XI achieves baseline-level accuracy with an 10% reduction in carbon footprint. CarbonCLIP-XS achieves an 8% increase in accuracy with a 3% reduction in carbon footprint compared to CinCCL6-8 CarbonCLIP-L, CarbonCLIP-M. and CarbonCLIP-S all achieve sig- nificant reductions in carbon footprint compared to TinyCLIP-39MM
with CarbonCLIP-I achieving 4% increase in accuracy and 4.5% reduction in carbon footprint, CarbonCLIP-M achieving an 1 11% reduction in carbon footprint with a 3% decrease in accuracy, and CarbonCLIP achieving a 17% reduction in carbon footprint without any regression in accuracy.
In terms of hardware configuration, CarbonCLII models select accelerators with cores of PF dimension 256 align with the underlying operator dimensions of the CLIP ViT-B/16 architecture, with sequence length of 197 for the vision encoder. Smaller models select total of 1024 PE units per core, whereas larger models select twice as many PEs to keep the latency of the task low. Due to the reduced size of the CarbonCLIP models, a 64KB local memory and 2MB global memory are sufficient keep the core utilized.
4.5 Evaluating for Different Compute Constraints
We evaluate CarbonCLIP models under three peak compute constraints inspired by real-world edge devices: 20 TOPS HA123 Nvi25), 4 TOPS [Goo20), and TOPS (WLS*24 In(25]. Figure Shows the resulting Pareto frontiers.
shows
Takeaway 6: As hardware shrinks due to lighter compute constraints, optimizing for energy (opera- tional carbon) increasingly minimizes both total carbon and latency.

---

Smaller hardware designs reduce embodied carbon, making operational energy the dominant factor the total carbon footprint. As result, mergy-optimizzed designs often achieve outcomes similar to those optimized for total carbon under tight compute budgets. Appendix [xxplorss the embodied vs. operational carbon trade-offs in more detail. Note that different grid intensities and expected hardware lifetime can also affect the ratio of embodied and operational carbon, affecting the optimization results, and the quality of each metric immensely, we show this Appendix K However, smaller accelerators have limited compute and memory. which increases inference velay-and thus total energy use use-despite lower power consumption. This leads to higher total carbon emissions some cases, underscoring the importance of hardware-monn performance. We further extended the study to evaluate the impact of latency constraints on model and hardware configurations, as well as their carbon footprints in Appendix
co-design to balance carbon efficiency and
5 Impact Discussion and Limitations
This work demonstrates a practical path toward sustainable Al by jointly optimizing model and hardware design for carbon reduction. Our framework is particularly effective for low-latency edge inference (c.g., chatbots, AR/VR) and highlights the potential for cross-domain collaboration across ML, hardware, and sustainability.
CATransformers currently targets Transformer models and domain-specific edge accelerators, with limited GPU support and evaluation CNNs. Extending other model families (e.g.. CNNs, Mamba) requires profiling the latency and energy of their unique operators and adapting pruning strategies beyond attention heads. Expanding to GPUs and data center scenarios is feasible, but the larger design space would demand significantly more search. Nonetheless, our GPU-based results (Appendices) L and M) suggest scalability training workloads and datacenter-level accelerators, where support for parallelization and distributed training could further improve carbon efficiency.
Beyond inference and training emissions, broader sustainability impacts, such electronic waste, water usage, and rare mineral consumption, remain underexplored. While our optimizations can implicitly reduce chip area and resource use, standardized metrics and ine-grained data currently lacking, limiting oulti-objective optimization over broader environmental impacts. Moreover, custom accelerators raise concerns about hardware heterogeneity and e- e-watte, which may be mitigated by reusing existing hardware or co-optimizing with commercial accelerators.
Proxy-based accuracy estimation, such as the methods employed in CATransformers, enables scalable exploration but may deviate from real outcomes due to overfitting or underestimating pruning effects. Developing more robust accuracy predictors is important direction for improving reliability.
6 Conclusion
We introduced CATransformers, a framework for co-optimizing model architectures and domain- specific accelerators minimize carbon footprint. By jointly considering operational and embodied carbon, CATransformers supports environmentally conscious design, particularly in edge computing. We demonstrated its effectiivness across various Transformer-based models moncll--iccluding multi-modal models--showcasing substantial potential in carbon reductions without sacrificing performance. This work fills key gap in sustainable AI deployment and provides foundation for future research in carbon-aware machine learning.
7 Acknowledgements
would like to thank Bernie Beckerman, David Eriksson, and Max Balandat for their expertise and assistance in building the optimization platform with Ax, Igor Fedorov for sharing insights on pruning models, Daniel Li and Hu Xu for providing valuable guidance on training CLIP models and working with the MetaCLIP dataset. We also would like thank Kim Hazelwood, Kristen Lauter, and Edith Beigne for supporting this work.
10

---

References
[ALK*23] Bilge Acun, Benjamin Lee, Fiodar Kazhamiaka, Aditya Sundarrajan, Kiwan Maeng. Manoj Chakkaravarthy, David Brooks, and Carole-Jean Wu. Carbon dependencies in datacenter design and management. SIGENERGY Energy Inform. Rev., 3(3):21-26, October 2023.
24] Muhammad Adnan, Amar Phanishayee, Janardhan Kulkarni, Prashant J. Nair, and Divya Mahajan. Workload-aware hardware accelerator mining for distributed deep learning training, 2024.
[App21] Apple Inc. Apple environmental progress report 2021. https://www apple. com/ http:////a..p.p.peaa...eeeaaaaaa Rarp pdf 2021.
Report__221_111
[BBL23] Lucía Bouza, Aurélie Bugeau, and Loic Lannelongue. How to estimate carbon footprint when training deep learning models? a guide and review. Environmental Research Communicaations, 5(1 15014, nov 2023.
[bJRE+23] colby banbury, Vijay Janapa Reddi, Alexander Elium, Shawn Hymel, David Tischler, Daniel Situnayake, Carl Ward, Louis Moreau, Jenny Plunkett, Matthew Kelcey, Mathijs Baaijens, Alessandro Grande, Dmitry Maslov, Arthur Beavis, Jan Jongboom, and Jessica Quaye. Edge impulse: An mlops platform for tiny machine learning. In D. Song, M. Carbin, and T. Chen, editors, Proceedings of Machine Learning and Systems, volume 5, pages 254-268. Curan, 2023.
20] Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, An- drew G Wilson, and Eytan Bakshy. Botorch: A framework for efficient monte-carlo bayesian optimization. Advances neural information processing systems, 33:21524- 21538, 2020.
[CHY+21] Kanghyun Choi, Deokki Hong. Hojae Yoon, Joonsang Yu, Youngsok Kim, and Jinho Lee. Dance: Differentiable accelerator/network. co-exploration. In 2021 58th ACM/IEEE Design Automation Conference (DAC), pages 337-342, 2021.
[CKES16] Yu-Hsin Chen, Tushar Krishna, Joel S Emer, and Vivienne Sze. Eyeriss: An energy- efficient reconfigurable accelerator for deep convolutional neural networks. IEEE journal of solid solid-state circuits, 52(1):127-138, 2016.
244 Benoit Courty, Victor Schmidt, Sasha Luccioni, Goyal-Kamal, MarionCoutarel, Boris Feld, Jérémy Lecourt, LiamConnell, Amine Saboni, Inimaz, supatomic. Mathilde Léval, Luis Blanche, Alexis Cruveiller, ouminasara, Franklin Zhao, Aditya Joshi, Alexis Bogroff, Hugues de Lavoreille, Niko Laskaris, Edoardo Abati, Douglas Blank, Ziyao Wang, Armin Catovic, Mare Alencon, Michat Stęchly, Christian Bauer, Lucas Otávio N. de Araújo, JPW, and MinervaBooks. mlco2/codecarbon: v2.4.1, May 2024.
[DBB21] Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Parallel bayesian opti- mization of multiple noisy objectives with expected hypervolume improvement. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS 22, Red Hook, NY, USA, 2021. Curran Associates Inc
[DBK+21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net, 2021.
[DCLTI9] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- training of Deep Bidirectional Transformers for Language Understanding, 2019.
[DGC24] Dayou Du, Gu Gong, and Xiaowen Chu. Model quantization and hardware acceleration for vision transformers: A comprehensive survey, 2024.
11

---

[ECA 23] Mariam Elgamal, Doug Carmean, Elnaz Ansari, Okay Zed, Ramesh Peri, Srilatha Manne, Udit Gupta. Yeon Wei, David Brooks, Gage Hills, et al. Carbon-efficient design optimization for computing systems. In Proceedings of the 2nd Workshop on Sustainable Computer Systems, pages 1-7, 2023.
[ECA *25] Mariam Elgamal, Doug Carmean, Elnaz Ansari, Okay Zed, Ramesh Peri, Srilatha Manne, Udit Gupta, Yeon Wei, David Brooks, Gage Hills, and Carole-Jean Wu. CORDOBA:
Carbon-Efficient Optimization Framework for Computing Systems. In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA), pages 1289-1303, Los Alamitos, CA, USA, March 2025. IEEE Computer Society.
[FKW 24] Ahmad Faiz, Sotaro Kaneda, Ruhan Wang. Rita Osi, Prateek Sharma, Fan Chen, and Lei Jiang. I.mearbon: Modeling the end-to-end carbon footprint of large language models, 2024.
[Fu24] Yao Fu. Challenges in deploying long-context transformers: A theoretical peak perfor- mance analysis, 2024.
[GDJ+24] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and et al. The Ilama 3 herd of models, 2024.
[GEH+22] Udit Gupta, Mariam Elgamal, Gage Hills, Gu-Yeon Wei, Hsien-Hsin S. Lee, David Brooks, and Carole-Jean Wu. Act: designing sustainable computer systems with an architectural carbon modeling tool. In Proceedings of the 49th Annual International Symposium on Computer Architecture, ISCA '22, page 784-799, New York, NY, USA, 2022. Association for Computing Machinery.
[GHL+25] Shreyank N. Gowda, Xinyue Hao, Gen Li, Shashank Narayana Gowda, Xiaobo Jin, and Laura Sevilla-Lara. Watt for what: Rethinking deep learning's s energy-performance relationship. In Computer Vision ECCV 2024 Workshops: Milan, Italy, September 29-October 4, 2024, Proceedings, Part XXII, page 388-405, Berlin, Heidelberg, 2025. Springer-Verlag.
[GIF+23]
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannanch Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets, 2023.
[GKL+22] Udit Gupta, Young Geun Kim, Sylvia Lee, Jordan Tse, Hsien-Hsin S. Lee, Gu-Yeon Wei, David Brooks, and Carole-Jean Wu. Chasing carbon: The elusive environmental footprint of computing. IEEE Micro, (2(4):37-47,J)) July 2022.
[GKX+24] Soroush Ghodrati, Sean Kinzer, Hanyang Xu, Rohan Mahapatra, Byung Hoon Ahn, Dong Kai Wang, Lavanya Karthikeyan, Amir Yazdanbakhsh, Jongse Park, Nam Sung Kim, and Hadi Esmaeilzadeh. Tandem processor: Grappling with emerging operators neural networks. ASPLOS. 2024.
[Goo20] Google. Helping you bring local ai to applications from prototype production, 2020.
[HA125] HAILO. Hailo-8 ai accelerator, 2025.
[Har25] Harmony Healthcare IT. Are you addicted to your phone? american phone usage & screen time statisties. https::/// ://p:://www http::/wwwwhhaaonhh.. harmonyhit. wttpiiit..cooooooooeee cww/phone - time-sta tistics/ 2025.
12

---

[HYW*22] Shiqi He, Qifan Yan, Feijie Wu, Lanjun Wang, Mathias Lécuyer, and Ivan Beschastnikh. Gluefl: Reconciling client sampling and model masking for bandwidth efficient feder- ated learning. In D. Song, M. Carbin, and T. Chen, editors, Proceedings of Machine Learning and Systems, volume 5, pages 695-707. Curan, 2023.
[HZRS16] Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[MME25] IMEC. imec.netzero. https:
imec- int. com/" 2025.
[ln225] Intel. Intel movidius myriad x vision processing unit, 2025.
[IWW+21] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Han- naneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. If you use this software, please cite it below.
[JKL Norman P. Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Cliff Young, Xiang Zhou, Zongwei Zhou, and David Patterson. Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings, 2023.
[JYP+17] Norman Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre-luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati. William Gulland, Robert Hagmann, C. Richard Ho, Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexander Kaplan, Harshit Khaitan, Daniel Killebrew, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson. Bo Tian, Horia Toma. Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, and Doe Hyun Yoon. In- datacenter performance analysis of tensor processing unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture, ISCA '17, page 1-12, New York, NY, USA, 2017. Association for Computing Machinery.
[JYS+20] Weiwen Jiang, Lei Yang, Edwin Hsing-Mean Sha, Qingfeng Zhuge, Shouzhen Gu, Sakyasingha Dasgupta, Yiyu Shi, and Jingtong Hu. Hardware/software co-exploration of neural architectures. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 39(12):805-4415 2020.
[KMH 20] Jared Kaplan, Sam McCandlish, Tom Henighan. Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
[Kri09] Alex Krizhevsky. The cifar- 10 dataset, 2009.
[LA22] LAION-AL. Clip benchmark, 2022.
[LBL 24] Haokun Lin, Haoli Bai, Zhili Liu, Lu Hou, Muyi Sun, Linqi Song, Ying Wei, and Zhenan Sun. Mope-clip: Structured pruming for efficient languael models with module-wise pruning eπror metric. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 27360-27370, 2024.
[LGH+24] Sasha Luccioni, Boris Gamazaychikov, Sara Hooker, Régis Pierrard, Emma Strubell, Yacine Jernite, and Carole-Jean Wu. Light bulbs have energy ratings- so why can't an't ai chatbots? Nature, 632:736-738, 08 2024.
13

---

[LIGT24] Baolin Li, Yankai Jiang, Vijay Gadepally, and Devesh Tiwari. Sprout: Green generative AI with carbon-efficien LLM inference. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pages 21799-21813, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
[LJS244 Sasha Luccioni, Yacine Jemite, and Emma Strubell. Power hungry processing: Watts driving the cost of ai deployment? In Proceedings of the 2024 ACM Conference Fairness, Accountabiliry, and Transparency, FAccT '24, page 85-99, New York, NY, USA, 2024. Association for Computing Machinery.
[LLSD19] Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quan- tifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700 2019.
[LMB + 15] Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft CoCO: Common objects in context, 2015.
[LYH21] Yujun Lin. Mengtian Yang, and Song Han. Naas: Neural accelerator architecture search. In 2021 58th ACM/IEEE Design Automation Conference (DAC), pages 1051-1056, 2021.
[Map25] Electricity Maps. The leading electricity grid api, 2025.
[MP24] Inc. Meta Platforms. Adaptive experimentation platform, 2024.
[MPA + 16] Divya Mahajan, Jongse Park, Emmanuel Amaro, Hardik Sharma, Amir Yazdanbakhsh, Joon Kyung Kim, and Hadi Esmaeilzadeh. Tabla: A unified template-basee based framework for accelerating statistical machine learning. In 20/6 IEEE International Symposium on High Performance Computer Architecture (HPCA), pages 14 26. IEEE, 2016.
[NSY24] NSYS Group. Average device lifespan: How long does a cell phone last? https //nsysgroup. wwt.cccmmclll..ee blog/ aveeraeeddvicce -ileice-llfeepann how wwwwwloog--oo does- cel 1 1-phone- last| 2024.
[Nvi25] Nvidia. Jetson orin nano, 2025.
[ONFL23] MossnmadHossein Olyaiy, Christopher Ng. Alexandra Fedorova, and Mieszko Lis. Sunstone: A Scalable and Versatile Scheduler for Mapping Tensor Algebra on Spatial Accelerators. In 2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2023.
[PMK+22] Amar Phanishayee, Divya Mahajan, Janardhan Kulkarni, Miguel Castro, and Muham- mad Adnan. Workload-aware hardware architecture recommendations, 2022. US Patent App. 17/965,681.
[PMT25] Amar Phanishayee, Divya Mahajan, and Jakub Michal Tarnawski. Integrated hardware architecture and distribution strategy optimization for deep learning models, 2025. US Patent App. 18/
[PRS+19) 19] Angshuman Parashar, Priyanka Raina, Yakun Sophia Shao, Yu-Hsin Chen, Victor A. Ying, Anurag Mukkara, Rangharajan Venkatesan, Brucek Khailany, Stephen W. Keck- ler, and Joel Emer. Timeloop: A Systematic Approach to DNN Accelerator Evaluation. In 2019 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pages 304-315, 2019.
[PSM+17] Jongse Park, Hardik Sharma, Divya Mahajan, Joon Kyung Kim, Preston Olds, and Hadi Esmaeilzadeh. Scale-out acceleration for machine learning. In Proceedings of the 50th Annual IEEF/ACM International Symposium on Microarchtecture, pages 367-381, 2017.
14

---

[3] Xinchi Qiu, Titouan Parcollet, Javier fernandez-Marquee Pedro P. B. Gusmao, Yan Gao, Daniel Beutel, Taner Topal, Akhil Mathur, and Nicholas D. Lane. A first look into the carbon footprint of federated learning. J. Mach. Learn. Res., 24(1), January 2023.
[RKH*21] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sand- hini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[SBV+22]
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wight- man, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, Patrick Schramowski. Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models, 2022.
[SDDN20] Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. Poor man's BERT: smaller and faster transformer models. CoRR, abs/2004.03844 2020.
[SGM20] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy consid- erations for modern deep learning research. Proceedings of the AAAI Conference on Artificial Intelligence, (3999)9336333136999 Apr. 2020.
[0] Ananda Samajdar, Jan Moritz Joseph, Yuhao Zhu, Paul Whatmough, Matthew Mattina, and Tushar Krishna. A systematic methodology for characterizing scalability of dnn accelcrators using scale-sim. In 2020 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pages 58 68. IEEE, 2020.
[SSL223] Chirag Sakhuja, Zhan Shi, and Calvin Lin. Leveraging domain information for the efficient automated design of deep learning accelerators. In 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 287-301, 2023.
[3] Dachuan Shi, Chaofan Tao, Ying Jin, Zhendong Yang, Chun Yuan, and Jiaqi Wang. UPop: Unified and progressive pruning for compressing vision-language transformers. In Proceedings of the 40th International Conference on Machine Learning, volume PMLR, 2023.
202, pages
[SVB+21] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of elipefiltered 400 million image-text pairs ArXiv, abs/2111.022 14, 2021.
[SWL SWL++2] H. Ekin Sumbul, Tony F. Wu, Yuecheng Li. Syed Shakib Sarwar, William Koven, Eli Murphy-Trotzky, Xingxing Cai, Elnaz Ansari, Daniel H. Morris, Huichu Liu, Doyun Kim, Edith Beigne, Reality Labs, and Meta. System-level design and integration of prototype ar/vr hardware featuring a custom low-power dnn accelerator chip in 7nm technology for codec avatars. In 2022 IEEE Custom Integrated Circuits Conference (CICC), pages 01-08, 2022.
[SZW+18] 18] Ananda Samajdar, Yuhao Zhu, Paul Whatmough, Matthew Mattina, and Tushar Krishna. Scale-sim: Systolic enn accelerator simulator. arXiv preprint arXiv:/811.0220 102883, 2018.
[vMVJ22] Jakub Žádník, Markku Makitalo, Jarno Vanne, and Pekka Jääskeläinen. Image and video coding techniques for ultra-low latency. ACM Comput. Surv., 54(11s), September 2022.
[VPF+24] Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, and Oncel Tuzel. Mobileclip: Fast image-text models through multi-modal reinforced training. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 15963-15974 2024.
15

---

[WDS* 19] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's s transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771, 2019.
[WES19] Yannan Nellie Wu, Joel S. Emer, and Vivienne Sze. Accelergy: An Architecture- Level Energy Estimation Methodology for Accelerator Designs. In 2019 IEEF/ACM International Conference on Computer-Aided Design (ICCAD), pages 1-8, 2019.
24]+ Tony F. Wu, Huichu Liu, H. Ekin Sumbul, Lita Yang, Dipti Baheti, Jeremy Coriell, William Koven, Anu Krishnan, Mohit Mittal, Matheus Trevisan Moreira, Max Wauga- man, Laurent Ye, and Edith Beigné. 11.2 3d integrated prototype system-on-chip for augmented reality applications using face-to-face wafer bonded 7nm logic at< 2,m pitch with up to 40% energy reduction at iso-area footprint. In 2024 IEEE International Solid-State Circuits Conference (ISSCC), volume 67, pages 210-212, 2024.
[WNM23] Irene Wang, Prashant J. Nair, and Divya Mahajan. FLuID: Mitigating stragglers in federated learning using invariant dropout. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[WPZ+23] Kan Wu, Houwen Peng, Zhenghong Zhou, Bin Xiao, Mengchen Liu, Lu Yuan, Hong Xuan, Michael Valenzuela, Xi (Stephen) Chen, Xinggang Wang. Hongyang Chao, and Han Hu. Tinyclip: Clip distillation via affinity mimicking and weight inheritance. In Proceedings of the IEÉE/CVF International Conference on Computer Vision (ICCV), pages 21970-21980, October 2023.
[WRG+22] Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang. Fiona Aga, Jinshi Huang, Charles Bai, Michael Gschwind, Anurag Gupta, Myle Ott, Anastasia Mclnikov, Salvatore Candido, David Brooks, Geeta Chauhan, Benjamin Lee, Hsien-Hsin Lee, Bugra Akyildiz, Maximilian Balandat, Joe Spisak, Ravi Jain, Mike Rabbat, and Kim Hazelwood. Sustainable ai: Environmental implications, challenges and opportunities. In D. Marculeseu, Y. Chi, and C. Wu, editors, Proceedings of Machine Learning and Systems, volume 4, pages 795-813, 2022.
[WSM* 18] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. CoRR, abs/1804.07461, 2018.
[WTPM24] Irene Wang, Jakub Tarnawski, Amar Phanishayee, and Divya Mahajan. Integrated hardware architecture and device placement search. In International Conference on Machine Learning, 2024.
[WWL+20] Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. Hat: Hardware-aware transformers for efficient natural language processing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2020.
[XXT+24] Hu Xu, Saining Xie, Xiaoqing Tan, Po- Huang, Russell Howes, Vasu Sharma, Shang- Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demysti- fying CLIP data. In The Twelfth International Conference on Learning Representations, 2024.
[ZDM*2) 22] Yanqi Zhou, Xuanyi Dong, Tianjian Meng, Mingxing Tan, Berkin Akin, Daiyi Peng. Amir Yazdanbakhsh, Da Huang, Ravi Narayanaswami, and James Laudon. Towards the co-design of neural networks and accelerators. In D. Marculescu, Y. Chi, and C. Wu, editors, Proceedings of Machine Learning and Systems, volume 4, pages 141-152, 2022.
[ZG23] Yiyang Zhao and Tian Guo. Carbon-efficient neural architecture search. In Proceedings of the 2nd Workshop on Sustainable Computer Systems, HotCarbon '23, page 1-7. ACM, July 2023.
16

---

[ZHS 22] Dan Zhang, Safeen Huda, Ebrahim Songhori, Kartik Prabhu, Quoc Le, Anna Goldie, and Azalia Mirhoseini. A full-stack search technique for domain optimized deep learning accelerators. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, page 27-42, New York, NY, USA, 2022. Association for Computing Machinery.
[ZLJG24] Yiyang Zhao, Yunzhuo Liu, Bo Jiang, and Tian Guo. CE-NAS: An end-to-end carbon- efficient neural architecture search framework. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
17

---

Appendix
A
Ablation Study: Pruning and Finetuning CLIP models
In this ablation study, we studied the effect of pruning each dimension and the effect of fine-tuning on various datasets to find a good proxy for approximating the accuracy after training the pruned models on larger datasets fir CLIP-based models. All models were evaluated against MS COCO. The results are shown in Figure 6 and our key observations are as follows:
We make some key observations in terms of the importance of each dimension to the overall accuracy of the model. First, accuracy drops significantly after pruning to 50% of any dimension, even after fine-tuning. Therefore, we confine the search space a minimum of half of each dimension. Second, the vision model has more significant impact on accuracy compared to the text model. Finally, pruning the embedding dimension has the most significant impact on accuracy among all pruned parameters, followed by number of layers, then FFN dimensions and number of attention heads.
Even with just a single epoch, fine-tuning on MS COCO significantly improves the accuracy of pruned models, making it a good and fast proxy for evaluating their overall potential. Fine-tuning
Fine-tuning on MS COCO:
Datacomp-Tings Training on small subset of a pre-training dataset (Datacomp-Tin) a 400k subset of Datacomp-Medium unfiltered) also improves accuracy, albeit with lower overall accuracy compared to MS COCO. Fine-tuning on Datacomp-Mediumt Using a general pre- pre-training dataset (Datacomp- Mediiu unfiltered) reduces the variance in accuracy between models of different sizes, showing that smaller models can achieve comparable accuracy when trained with enough data. This incentivizes our post-pni post-pruning training with large and high quality MetaCLIP 2.5B Dataset.
In general, larger models will attain higher accuracy compared to models with fewer parameters when trained for the same number of steps. However, models with fewer parameters may still achieve comparable accuracy as larger models given more training steps [ Therefore, we fine-tune each model with the same computation FL.OPS, allowing smaller models train for more FLOOP and recover their accuracy from more extensive pruning.
Loper
TTV
Taxl Erdeddinngggg
Drer
Vidon
Viton Srawecding
0.
Figure 6: Pruning and fine fine-tuning results for each dimension of the Text and Vision encoders.
B Estimation Tool-chain Integration
In this section we present in detail the integration of the tools used in the hardware estimator as illustrated in Figure 7 We use Accelergy [WES19] to estimate the area and access energy of each component, and Sunstone [ONFL23 to estimate the per-operator latency and energy. For carbon estimations, we use ACT |EE 22] to estimate the embodied carbon of the hardware architecture based on the area the accelerator. Given the energy estimates from Sunstone, Electricity Maps [Mpp55 is used to retrieve the carbon intensity of the electricity of a given grid location and calculate the operational carbon of executing a single inference. We then scale the operational carbon to the total lifetime of the hardware architecture.
18

---

Carbon Intensity estimation
Input
Tools
O
Intermediate Output
Output
Electricity Maps
Symbolic Trace
Model Config
Torch.FX
Operator graph
Per operator estimation
Operational
Energy
Carbon
Sunstone
Per architecture estimation
Latency
Access Energy
Embodied carbon estimation
HW Architecture Config
Accelergy
Embodied
Cacti
Area
ACT
Carbon
Figure 7: Integration of tools to estimate the total carbon footprint of the given hardware and model configuration.
Proxy (MSCOCO)
Final (Average
80
Accuracy (9)
40
Figure 8: Correlation between the proxy accuracy, fine-tuned and evaluated on MS COCO and the final model accuracy evaluated across 41 datasets for pruned CLIP models.
C Correlation Between Proxy and Final Model Accuracy
To efficiently approximate accuracy while preserving ranking consistency, we fine-tune each pruned model to a common dataset used to evaluate each model type. As shown in Figure 8 fine-tuning maintains high Spearman's 's rank correlation coefficient (0.98) with the final post-pruning training accuracy, ensuring reliable accuracy ranking. report the mean top-1 recall accuracy on MSCOCO the accuracy proxy for each pruned CLIP model. We incorporate this into the optimization loop to dramatically reduce the cost of evaluating candidate model architectures.
D ISO Accuracy
In this sections we present full details different design points found by each optimization metric for given accuracy point in Tables 4 6 and 7 for CLIP-ViT-B16, BertBase, Llama3-8b, ViT-B16 respectively. The key observations are discussed in Section 4.2 For each iso-accuracy comparison, accuracy values are reported with a tolerance of +/- 1 % across optimization strategies.
E Pareto Frontier of each Optimization
In this section, we present an example of Pareto frontier for each optimization metric using the CLIP VIT-B/16 model. Figur the Pareto frontiers for latency-only, carbon-only, energy-only, and caabno optimization objectives. Each data point in the figure represents specific model and hardware architecture configuration, with accuracy shown the y-axis, carbon footprint on the
19

---

Table 4: The hardware and model architecture configuration found by each optimization metric at each accuracy point for CLIP-ViT-BIG Hardware configurations are specified in the format of: {TC, pēz, pey, L2, L2b gib}. Text and Vision encoders are specified in the format of (Num Layers, FEN Dim, Hidden Dim, Num Heads)
Accuracy (+f 186)
Optimization Metric
Carhon (ggCO2e)
Latency (ms)
Hardware Configuration
Conliguration
Model
Text Encoder Configuration
Vision Encoder
Configuration
Params (M)
Carbon
12.6
256, 64,
[9. 1536, 6)
[12, 576. 768, 8]
Energy
0.50
3.9
[2, 256, 16, 128, 32, 2
(7. 1536, 384. 8)
[12. 576, 768, 8)
101
31%
Latency
0.55
12, 256,
[10. 1792, 384.
768, 11]
Carbon + Latency
0.48
8.8
[2, 256, 8, 64, 64, 2]
(9. 1280, 512, 6)
[11. 672. 768. 9)
105
Carbon
10.9
1,25,
64
(9.
63
[11. 768.6)
19.5%
Enorgy
0.48
3.5
[2, 256,
16, 128, 32, 21
(6. 1536, 512, 8)
[12, 480, 768, 7)
90
I Latency
0.55
8.2
256,
64,,
21
[11, 1536, 384,
576, 768. 12]
Carbon Latency
0.45
7.3
[2, 256,
4, 64, 128, 21
(9. 1024, 512, 5)
(11, 768. 8)
94
Carbon
0.43
22.1
256,
64,
2]
[7. 1536, 6384
576, K]
13%
Energy
0.49
7.3
[2, 256, 64, 64, 2]
(8, 1792, 448, 5)
(10, 576, 768, 7)
92
Latency
0.54
15.9
256,
128, 64,
[12. 1792, 320,
(11. 576. 768. 12]
96
Carbon Latency
0.47
7.3
11. 256, 16,
24
(9, 2048, 384, 4)
3072, 768,
Carbon
0.32
4.6
[1, 256, 8, 24,
(6, 1024, 256, 4]
[6. 1536, 384, 6)
27
Energy
1.8
[1. 256, 16,
(6. 1024, 256, 6)
[6. 1536, 384, 6]
2.5%
Latency
0.46
1.3
[4, 256, 16, 128, 128, 2]
(6. 1024, 384. 8)
(9. 1536, 384, 4)
43
Carbon Latency
0.31
256,
[6. 1536,
Table Iso-Accuracy points for Bert-Base model. Original model configuration: number of layers:
12, embedding dimension: 768, intermediate dimension:3072, number of Attention Heads:12
Accuracy
Optimization
Carbon
Latency
Hardware Configuration
Model
(+/- 1%)
Metric
(kgCO2e)
(ms)
Configuration
Params (M)
Carbon
0.26
12.7
[1, 8, 64, 32,
[10. 192, 48,
Energy
26
2.9
[4, 1, 62, 64, 64, 2]
[6. 384, 48,
1.7
63%
Latency
0.41
0.9
[4, 256, 8, 256, 128, 4]
(9.192.144.6)
5.5
Carbon + Latency
0.31
0.9
[4.4, 128. 8, 128, 64, 2)
[8. 576, 48, 4)
1.9
Carbon
0.44
10.9
128. 64, 64.
[7, 48. 12)
1.9
Energy
0.48
3.5
[1. 128, 2, 128, 128)
[8. 1344, 48,
2.5
68%
Latency
0.55
8.2
256, 16, 128, 128, 4]
[7. 1920. 48, 6)
2.7
Carhon + Latency
0.45
7.3
[2, 256, 16, 256, 128, 2]
[6, 3072, 96, 8)
6.4
Table 6: Iso-Accuracy points for Llama2-7b model. Original model configuration: number of layers:
32, intermediate dimension: 1008, embedding dimension: 4096, number of Attention Heads:32
Accuracy
Optimization
Carbon (kgCO2e)
Latency
Hardware Configuration
Model
(+/- 1%)
Metric
(ms)
Configuration
Params (M)
Carbon
0,26
12.7
[1, 8, 32, 2]
1376. 256, 32]
38.2
Energy
26
2.9
(1,64, 8, 64, 128, 4)
(440, 256, 32]
41.2
64%
Latency
0.41
0.9
[4, 8, 128, 128, 4)
1376, 256. 32)
38.2
Carbon Latency
0.31
0.9
[4, 128, 16, 64, 128,
1376, 256, 32)
38.2
Carbon
0.44
10.9
[1,64, 8, 64, 64,
1376,,, 256, 32]
Energy
0.48
3.5
[4, 128, 16, 128, 64,
1376, 256, 32)
41.8
66%
Latency
0.55
8.2
256, 16, 512, 64, 8)
(128,28, 256, 32]
42.4
Carbon + Latency
0.45
7.3
(1. 256, 512, 128, 4)
1376, 256. 32)
41.8
Table 7: Iso-Accuracy points for ViT-B-16 model. Original model configuration: number of layers:
12, intermediate dimension:3072, embedding dimension: 768, number of Attention Heads:12
Accuracy
Optimization
Carbon
Latency
Hardware Configuration
Model
(+/- 1%)
Metric
(kgCO2e)
(ms)
Configuration
Params (M)
Carbon
0.26
12.7
[1,32 32, 64, 64, 2)
[12, 1920, 2496, 6]
45.8
Energy
26
2.9
[28, 32, 64, 128, 2)
[11, 1344, 2496, 12)
31.7
39%
Latency
0.9
(2, 256, 32, 256, 128, 4]
(7,21 2688, 6)
31.7
Carbon + Lateney
0.31
0.9
[2, 256. 16,
128,
[8, 1728, 2496, 6)
29.0
Curbon
0.44
10.9
(2, 128, 128, 64
211, 2688,
49.2
Energy
0.48
3.5
[1. 216. 16, 64, 64
(12. 1728, 2496, 12)
39,5
40%
Latency
0.55
8.2
(2, 256, 16, 64, 64, 8)
[11, 1728, 2496, 8)
43,2
Carbon + Latency
0.45
7.3
[4, 64, 32, 64, 128,
[12, 1728, 2688, 8)
48.0
20

---

0.5
333
033
0.3
Boas
Bazs
8azs
0.0
630
Total
6.00
012
Carbon +
010
Lafency
Latency
43
01
be00
Figure 9: Pareto frontiers for different optimization modes under 20 TOPS compute constraint.
Table 8: Summary of Hypervolume indicator difference across runs of the same configuration
Standard Deviation Coefficient of Variation
Model
Metric
Hypervolume Mean
(%)
VIT-B/16
carbon
0.27
0.0006
0.2
latency
0.04
0.0001
0.3
energy
0.07
0.007
9.7
carbon latency
0.26
0.009
3,5
BertBase
carbon
0,50
0.001
0.2
latency
0,70
0.004
0.6
energy
0.65
0,003
0.6
carbon latency
0.48
0,013
2.8
Llama3
carbon
0.45
0.033
7.4
latency
0.66
0.004
0.6
energy
0.35
0.003
1.1
carbon lateney
0.43
0.003
0.7
CLIP
carbon
0.19
0.021
11.1
latency
0.18
0.016
9.2
energy
0.39
0.015
3.8
carbon latency
0.13
0.004
3.2
X-axis, and latency encoded via color map. To ensure consistency, each experiment is repeated three times, and accuracy is estimated using the MS COCO dataset. When latency is not an explicit optimization objective, maximum latency constraint of 50 ms is enforced [vMV]22 realistic deployment scenarios.
ensure
Consistent with the discussion in Section 4.2 examining the Pareto frontiers reveals that optimizing solely for total carbon yields configurations with the lowest carbon footprint across all accuracy levels. However, this comes at the cost of higher latency compared to the other optimization objectives.
F Consistency of Experimental Results
Each optimization is repeated three times for fair evaluation. To assess consistency, we compute the Hypervolume (HV) indicator across runs with the same configuration. HV measures the portion of the objective space dominated by the Pareto front, relative a reference point. Across all model- hardware optimizations, the standard deviation of HV remains below 0,03, with an average coefficient of variation (std/mean) under 3.5%, indicating stable and reproducible results. Table summarizes these statistics for each model architecture and optimization objective.
G CLIP VIT B/32 Accuracy Evaluations
In this section, we present results that demonstrate the applicability of CATransformers to the CLIP ViT-B/32 architecture and compare performance against the TinyCLIP baselines. (Table Our findings show that CATransformenss optimization can be effectively generalized to other model architectures, yielding up to 5% and 8% reductions carbon footprint while achieving higher accuracy and comparable latency compared to the TinyCLIP baselines, respectively.
21

---

Table 9: The hardware and model architecture properties of each variant of the CarbonCLIP-BAN family. Hardware configurations are specified as: {TC, PEg. PEy. L2, 12b GLB). Text and Vision encoders are specified as: (Num Layers, FFN Dim, Hidden DIm, Num Heads)
Carbon (kcCO2c)
Latency (ms)
Hardware Configuration
Model Conliguration
Avg. Accuracy
Name
Text Encoder Conliguration
Vision Encoder
over 41 datasets
(M)
Configuration
CLIP-R/32- DaraComp
0.42
15.1
12,
64.21
(12. 2048, 512. 8]
[12, 3022, 768,
144
TinyCII--9MM/
0.37
3.0
[1.
32, 52,
64,
32,
[6, 2048, $12, 8)
[12, 2048, 512. 8)
84
45.2
ThnyCI MP/26
0.39
9.4
12128,
64,
61
2048, $12,
2560, 640,
115
47.2
CarbonU LIP-32--
0.35
7,3
[1,64, 16,
64,21
[6. 1536, 384, 5)
[TO, 3072, 672. 12)
89
46.4
CarbonC1.1P-15
0.36
15.3
[1.64,
64, 64,
1280, 448,
[11, 2688, 768,
47.6
CarbonCT.PP-32-1
0.38
9.1
[1, 128,
8, 64, 128,
(7.2048,512.6)
(11,3072,768,10) 10)
115
49.1
Table 10: Results across all 41 evaluation benchmarks from CLIP Benchmark
Cartont 1.P
DataCompp B/G6
C.r 1732
TI B33
DaraCompp R732
cus
0.57
0332
0.14
0.16
0.17
0.3
0.18
0.15
0.15
0.15
6=20133
0.17
0.21
0.3
0.36
0.34
0.52
0.53
0.25
0.23
0.49
fgve airmatt
8.17
0.2
0:35
0:35
0.29
0:35
0.00
0.2
0.21
0.14
0.18
dickk200
0.55
0.65
0.7
0.7
0.76
0.76
0.52
0.76
0.65
0.67
0.68
0.68
0.71
0.7
flickeik
0.52
0.6
0.65
0.64
0.7
0.71
0.5
0.7
0.62
0.64
0.56
0.63
0.56
0.65
gloab
0.25
0.6
0.46
0.5
0.55
0.32
0.11
0.55
0.47
0.49
0.52
0.35
0.3
0.52
imagreat-a
0.11
0.19
0.26
0.32
0.3M
0.35
0.15
0.48
022
0.22
0.24
1.17
1.21
U.S
imagenet-
0.35
0.25
0.51
0.44
0.44
0.40
0.4
0.43
0.29
0.49
0.5
0.52
0.51
0.5
imagenet.
0.35
0.65
0.73
0.37
0.52
0.7
0.3
0.54
0.72
0.74
0.75
0.7
0.73
0.75
imsgenal 11
0.51
0.6
0.63
0.65
0.7
0,63
0.41
0.74
0.62
0.64
0.65
0.6
0.62
0.69
dortch
0.35
0.55
05
0.52
0.57
0.99
U.51
0.53
0:11
0.35
0.56
0.62
0.56
0.35
0.54
0.95
0.54
0.58
0.66
0.69
0.75
0.20
D.45
0.46
0.54
0.6
0.5
0.11
s010
094
0.66
097
0.98
097
0.5
0.96
0.5
D.66
0.99
0m6
0:68
0.69
0.71
0.69
0.56
0.6
065
067
0.23
0.35
0.77
8.79
0.77
0.62
0.77
0.77
0.77
678
031
3o27 multisaell
0:25
0.29
0:31
0:31
0.3
0.22
0.74
0.1
0.79
0K
0.8
1.79
0.79
vabicaleech10
0.5
0.33
0.34
0.84
0.85
0.82
0.72
0.55
0.3
0.33
0.55
0.2
082
0.44
vabiiiitul
0.73
0333
0.39
09
0.93
0.1
0.73
0.9
0.9
0.92
182
0.6
vubiiifat100
0.47
0.55
0.65
0.67
0.75
0.68
0.42
0.52
0.99
0.7
0.73
0.00
0.72
08
vussccess clesest object distance
0.16
0:15
0.17
0.06
0.1M
012
0.16
0.24
U.16
0.16
0.16
0.17
0.21
0.21
vossillur coentt all
8:18
8.19
0:31
0:14
0:38
0.11
0.15
B16
D.2
0.35
0.24
0.1%
vulxtiabetic_eetiopppc retioputyy
0.04
0.39
0.16
0.04
0.2
0.03
0.00
0.11
BJTT
0.05
0.1
0.24
0.42
vabikmlle
0.14
0.15
0.15
0.15
0.14
0.13
0.16
0.19
021
0.2
0.14
0.21
0.15
0.16
yabllsprioo..ae Jabel
Liniiiiiiio.blee_miiaaaiaa Liiietation
0.02
0.02
0.02
0.02
0.02
0.02
0.00
0.02
0.02
0.04
0.03
0.02
0.02
0.05
vaistigriio linel positon
001
003
005
0.8
0.01
0.08
0.00
001
002
0,03
vaisthd
vablcusual
0.30
0.59
0.58
0.
0.5
0.45
0.71
vubiiini wlosesttttelllaa vewiiceeeeilleeeeeee vesuddeediauuee distunce
0.37
0.32
0.3
0.26
0.35
0.11
0.25
0.19
0.17
vibippnn
063
0.6
0.61
0.5
0.1
0.58
vabipes
0.76
0.85
0.37
0.39
0.91
0.11
0.46
0.87
0.88
0.8
vatmitetts
8:65
0.57
0A3
0/4
0.56
0.35
9.31
0.8
US5
0.55
US4
BS6
UA3
vebienalizeh kinel awimmth
800
0.06
0/06
0.66
0.08
0.06
0.66
003
003
vublknallzoch eievaato
8.11
0.11
0.12
0.31
0.11
0.1
0.12
0.11
811
ILI
0.11
0111
811
81
vablesha
0.15
0.3
0.29
0.39
0.38
0.16
0.14
0.61
0.3
0.45
0.42
0.33
0.50
0.61
H CLIP Benchmark Full Result
In this section we provide a detailed breakdown on the accuracy of each dataset for CarbonCLIP and the evaluated baselines in Table 10
I Carbon Footprint Breakdown
We provide a breakdown of the carbon footprint for each variant of the CarbonCLIP family model. As shown in Figure [0] as the model increases in size, the proportion of operational carbon in the overall carbon footprint of the model increases from 20% to over 40%. The CarbonCLIP-XX model has 3 x the number of parameters and almost 3 x the latency of CarbonCLIP-XX but the selected hardware architecture only has double the number of compute PEs. Therefore, the operational carbon increases proportionally more than the increase in embodied carbon. This highlights need of co-optimizing the model and hardware architecture to maintain an intricate balance between operational and embodied carbon, keeping the overall carbon footprint of the system low. As such, the expected lifetime and source of power are also important factors that need be taken into account during the optimization process.
22

---

Operational (totai lifetime)
Embodied
100%
Carbon Breakdown
75%
25%
0%
XS
XL
Average Accuracy (%)
Figure 10: Operational and embodied carbon footprint breakdown for the CLIP-ViT-B/16 arw-viiiBBBlaachhteeeuuur
0.40
carbon-100ms
carbon-50ms
0.35
carbon-10ms
Accuracy (96)
0.25
0.20
0.15
0.10
0.05
0.00
0.3
0.4
0.5
0.6
0.7
0.8
Carbon (kgCO2e)
Figure 1I: Latency and carbon footprint trade-offs for the CLIP-ViT-B/16 architecture across the
Pareto frontier.
Case Study: Varying Latency Constraints
We evaluate the impact of latency constraints model and hardware configurations, well as their carbon footprints. We categorize use cases into three categories based on latency requirements: critical real-time (< 10ms), interactive (<50ms), and non-critical (<100ms) [vMV]22] Figure shows the Pareto frontiers for each category.
Lower latency constraints typically result in less carbon-efficient designs. For example, 10ms latency constraint achieves only an 17% carbon reduction compared to latency-optimial models, although with comparable latency values. However, increasing the constraint to 100ms does not consistently improve efficiency, as many optimal designs already meet the 50ms threshold.
K
Case Study: Varying Operational Regions
The region of the operation and manufacturing of the model affects the search results, and the quality of the metric immensely. Figure 12 shows that varying the operational carbon region, in high, mid, low carbon-intensity regions, yields dramatically different results in terms of the model searched and resulting configurations. Additionally, while our case studies (Section 4.5) show that energy optimization is effective when operational carbon dominates (e.g., smaller architectures, high-carbon- intensity regions), in cleaner energy regions, total earbon optimization still yields lower emissions. Optimizing against total carbon instead of energy can reduce emissions by 8% in low-carbon regions (Canada) vs. 2% high-carbon regions (Taiwan). The pareto frontier of each region shown in Figure 13a and Figure 13b
23

---

0.5
Taiwan
0.4
CA, USA
BC, Canada
0.3
Accuracy
0.2
0.1
0.0
0.4
0.5
Carbon (kaCO2e)
Figure 12: Pareto Frontier of Carbon-only Optimization searches across High (Taiwan), Mid (Califor-
nia, USA), and Low (British Columbia. Canada) carbon intensity regions.
0.5
0.5
Carbon
Carbon
0.4
Energy
0.4
Energy
Accuracy (%)
0.3
Accuracy
0.2
0.2
0.1
0.1
0..25
0.8.3
0.45
0.50
0.4
0.5
0.6
0.7
Carbon (kaCO2e)
Carbon (kqCO2e)
(a) British Columbia (Low carbon intensity)
(b) Taiwan (High carbon intensity)
Figure 13: Pareto Frontier of the Carbon vs Energy Optimization result in regions with different carbon intensity.
L Case Study: Model Architecture Search with GPUs
Once a hardware architecture is selected, CATransformers supports model architecture search on fixed hardware to refine configurations and re-evaluate carbon costs before manufacturing.
Table 11 shows example of model architecture search fixed V100-like hardware architecture, optimizing total carbon and latency.
M Energy and Latency Validation with GPUs
To maintain high confidence in our estimation toolchains, validated our energy and latency estimates using existing GPU hardware (V100, A100, and H100). Although GPUs differ from
Table 11: Model architecture search with fixed V100-like architecture. Text and Vision encoders are specified in the format of [Num Layers, FFN Dim, Hidden Dim, Num Heads)
Proxy Accuracy (%)
Carbon (kgCO2e) Latency (ms)
Text Model Config
Vision Model Config
2.8
11.13
37
6,1022,512,6
6, 6,136,667,,6 9,3072,768,6
9.8
11.41
58
6.2048,512,8
14.4
11.45
6,2048.512,6
10,2688,768.
25.3
11.57
76
6,1024,512.7
12,2688,766
28.7
11.58
76
6.1792.512.7
12,2688,7688
24

---

Estiation Messured
Lutency Scaled Lalency
0.04
0.05
Latency (9)
Enery
0.02
Explored Model Archurciu
Explored Mocel Architottures
(a) V100 Energy
(b) V100 Latency
Estimsted Erwryy Masoured Energy
Estimate Latandy Scaled Measured Lalenoy
2.5
0,0200
M
Eleg/dt
2.0
Laterey (4)
0.0150
1.5
0.0200
Explored Model tectures
Explored Model Archilectures
(c) A100 Energy
(d) A100 Latency
Energy Energy
Estimated Latency Scaled
Moasure Latency
0.0160
0.0125
Elergy 10
Latency (8)
0.0100
0.0050
Explored Modal Archtactures
Explored Model Aechitectures
(c) H100 Energy
(f) H100 Lateney
Figure 14: GPU estimated vs measure performance.
Table 12: Latency and Energy estimation error against Real GPU hardware
Hardware
Energy Estimation Error Spearman's (Knergy) Latency Estimation Error Spearman (Latency)
V100
0.97
7.4%
A100
12.3%
0.7
11.1%
Nomain--spccife accelerators, we modeled a GPU-like architecture with comparable tensor and vector units, conducted a model search on the fixed architecture, and profiled actual hardware performance for each searched data point.
The results demonstrate strong correlation between our estimates and measured performance, with Spearman's 's rank-order correlation ranging from 0.5 1.0. When energy and latency values are scaled to common measurement range using constants, the estimates yield average errors of 8% for energy and 9% for latency across GPU architectures. This validation confirms that our estimation tools provide accurate and reliable results, even with simulated hardware. Table 12 summarizes the results, while Figure 14 presents the estimated and measured performance each evaluation GPU architecture.
25

---

N Carbon Footprint of the Framework
We quantify the carbon footprint of running CATransformers using CodeCarbon [SLL*24] On average, the optimization process takes 5 hours for BERT and up to 20 hours for CLIP models. For CLIP, the most resource- resource-intensive intensive case, 100 optimization trials emit approximately 57 kgCOze, while final model training emits 454 kgCOze per model. This means the optimization process costs roughly 1/13th the carbon budget of training the final model. Despite the one-time cost of optimization, CATransformer achieves overall efficiency gains through reduced training steps post-pruning, along with inference gains that scale with the number of devices.
26

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
All claims made in the abstract and introduction are substantiated within the paper. An overview of the framework and its design details is provided in Section while key insights, analysis, and empirical results related to carbon optimization are presented in the evaluation sections, particularly in Section 4.2] Training results for the CarbonCLIP model and comparisons to prior work are discussed in Sections 4.3 and Additional
Justification:
implementation details and ablation studies are included the Appendix. Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We included a Discussion and Limitations (See Section $5)).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
27

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
Paper does not include theoretical proofs.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper provides detailed overview of all framework components in Section including search parameters and integration methodology. Additional information on the estimation toolchain is provided in Appendix Bot ensure full reproducibility. The experimental setup, covering evaluation baselines, dataset and model sources, as well as constants and assumptions used, is outlined in Section 4.1 Furthermore, the source code is included as supplementary material for reference.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
28

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The source code is included as part of the supplementary material and will be open-sourced upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/pu
blic/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.co wttpccccpppccccceeco... beciic/guiiee guides/ Codeeuomisssseeeeeeccee soinoooccc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental setup details, including all evaluated models, datasets used, and hyperparameters, are provided in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Spearman's rank correlation coefficients are reported to establish the statis- tical significance of fine-tuning proxies (Section 3.2 Appendix C and GPU validation
29

---

results (Appendix M. For the Bayesian Optimization results in Section each experi- ment repeated three times ensure consistency. We demonstrate the reproducibility of the optimization outcomes by evaluating the Hypervolume (HV) indicator across runs of the same configuration, and report the standard deviation and coefficient of variation for each model and optimization objective in Appendix F Estimation error rates for latency and energy are discussed in Section 4.1, with additional details provided in Appendix M Accuracy deviations relevant to ISO-accuracy comparisons are clearly reported in Figure in Section
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section 4.1 provides details for the compute platform, memory, and time
required for experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttps://eeuripp httpr//pp.p.ppppptttttttssaataie
Answer: [Yes]
Justification: Authors have reviewed the NeurlPS Code of Ethic, and all submitted material preserves anonymity.
Guidelines:
30

---

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss both the limitation and impact of the work in Section 5
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., dep.oymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
31

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The paper cites all relevant prior work, including frameworks, codebases, datasets, and pretrained models used in both the design and evaluation. Detailed references are provided in Section Section 4.1 and the Appendix.
Justification:
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com /ataastssss has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Source code is included as part of the supplementary material (anonymized), with detailed instructions on how to run and reproduce the experiments.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
32

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Core method development in this research does not involve LLMs as any important, original, or ntn-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy for what should or should not be described.
33


Reviews:
[{"review_id": "etb9tWQJyn", "reviewer": "Reviewer_oJBF", "summary": "The paper proposes a model-architecture and hardware co-design framework for jointly optimizing priority metric like carbon footprint (both embodied and operation) followed by latency, energy and accuracy. A new open source hardware evaluation tool chain for design space exploration is also showcased. The framework is evaluated across multiple directions with join hardware and model architecture improvement, ablations with hardware or model architecture fixed. The work provides a basis for important decision-making in choosing co-optimized hardware and model architecture for sustainability focused end-users.  Co-optimized model–hardware configurations reduce total carbon by 30% over latency optimized and 8% over energy-optimized baselines.", "questions": "* Based on the discussion in section 3.1: Does it mean that the carbon footprint of the pretrained model can't be optimized since we start from it and run the pruning. Or, is the suggestion that if we pretrain on the pruned model from scratch, we will recover the same accuracy? (this is in realation to the discussion in last line of 3.5)", "limitations": "None that I could identify since I am not an expert in co-design area.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "+ Good paper on framing the groundwork for identifyng design tradeoffs for large models\n+ Good use of light-weight fune tuning as a proxy for accuracy during Bayes Opt.\n+ The open-source toolchain is really useful for holistic design space exploration for latency, energy and total carbon emission\n+ It shows faster/high volume evaluation rate (100M/100iters)\n+ Pruned models require less training budget than full-pretraining.(60% reduction)\n+ The ablations and the corresponding takeaways also help gain great insights in to model architecture and hardware design choices,\n+ The works highlights the complex trade-offs that were not previously considered in earlier works\n\n- None that I can identify", "was_revised": true, "final_justification": "I would retain my current score.", "rebuttal": "We thank the reviewer for their thoughtful feedback and for recognizing the significance and quality of our work, noting that the paper “provides a basis for important decision-making…for sustainability-focused end-users” and “highlights the complex trade-offs that were not previously considered in earlier works.”\n\n**Question 1: Optimizing pretrained models**\n\nCATransformer takes a pretrained model as input and identifies pruned submodels and hardware architecture configurations that together minimize the total carbon footprint. We show that post-pruning training on the selected submodel can achieve accuracy comparable to the original model while significantly reducing carbon emissions (section 4.5 Table 3). Alternatively, given a fixed pretrained model, CATransformer can perform a hardware architecture search to identify the configuration that minimizes carbon footprint under specified latency constraints (Section 4.3 Table 2)."}, {"review_id": "Ty9WMCjxog", "reviewer": "Reviewer_Z64Y", "summary": "This paper introduces CATransformers, a novel co-design framework that jointly optimizes Transformer-based model architectures and hardware accelerators with carbon efficiency as a primary objective. It addresses the growing carbon footprint of AI systems by incorporating both operational and embodied carbon into the early-stage design space exploration. The framework integrates a multi-objective Bayesian optimizer, a structured pruning-based model evaluator with proxy fine-tuning, and a hardware estimator that quantifies carbon, latency, and energy. Experiments across language, vision, and multi-modal models demonstrate CATransformers’ ability to reduce total carbon emissions without sacrificing accuracy or latency, highlighting its effectiveness through the design of CarbonCLIP, a family of low-carbon CLIP variants.", "questions": "1. The current framework focuses solely on inference-time optimization — does this neglect the potentially much larger carbon cost of training, especially for large-scale pretraining?\n\n2. While the method reduces model carbon footprint, could the carbon cost of the search process itself be substantial and offset the benefits?", "limitations": "While the paper does offer some discussion of its limitations, I would suggest that it could be further strengthened by considering the following aspects.\n\n1. The authors should discuss how to balance the potential sustainability challenges introduced by custom accelerators, such as increased electronic waste and maintenance costs due to hardware heterogeneity.\n\n2. The authors could briefly discuss the risks of proxy-based estimations deviating from real-world outcomes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. This paper is the first to systematically incorporate carbon emissions as a primary optimization objective in the co-design of Transformer architectures and hardware accelerators. \n\n2. The proposed CATransformers is designed to be reproducible and adaptable across a wide range of Transformer models.\n\n3. The paper systematically analyzes trade-offs across multiple objectives, model scales, and hardware configurations.\n\n\nWeakness:\n\n1. Some components in the method section are described rather compactly, which may hinder reproducibility without the code release.\n\n2. The current framework exclusively targets inference optimization. It does not account for training-phase carbon cost, especially from large-scale pretraining of foundation models. And this may dominate the total carbon footprint. This limits the general applicability of the framework.\n\n3. While the method aims to reduce model carbon impact, the carbon cost of the search process itself is not measured or discussed.", "was_revised": true, "final_justification": "The authors provide more details and make more comparisons to refine the article. This article will make contribution to the society of research about carbon and environment.", "rebuttal": "We thank the reviewer for their thoughtful feedback and suggestions. Below, we answer the questions in order: \n\n**Weakness 1: Details of the methods and key components**\n\nWe will open-source the codebase upon acceptance and include detailed tutorials and examples to support reproducibility. In the final version of the paper, we will also expand the method section to provide additional implementation details.\n\n**Weakness 2, Question 1: Extensibility to training and datacenter accelerators**\n\nThis work focuses on making model inference carbon-efficient through hardware-model co-design. Carbon footprint of inference in aggregate is significant due to much wider usages [1]. We also agree with the reviewer that large-scale pretraining of foundation models can result in substantial carbon emissions. As our GPU-based results from Appendices L and M show, our proposed design methodology can scale to datacenter-level accelerators. CATransformer inherently supports training by modeling forward and backward pass latency and energy at the operator level. It can be extended to incorporate parallelization strategies and training system design, enabling optimization of carbon efficiency in distributed training.\n\n[1]  Luccioni, S., Jernite, Y. and Strubell, E., 2024, June. Power hungry processing: Watts driving the cost of ai deployment?. In Proceedings of the 2024 ACM conference on fairness, accountability, and transparency (pp. 85-99). \n\n\n**Weakness 3, Question 2: Carbon Footprint of the optimization process**\n\nWe quantify CATransformer’s carbon footprint using CodeCarbon [1] . On average, the optimization process takes 5 hours for BERTbase and up to 20 hours for CLIP. In the most resource-intensive case (CLIP), 100 optimization trials emit approximately 57 kgCO2e, while final model training emits 454 kgCO2e per model.  **This means the optimization process costs roughly 1/13th the carbon budget of training the final model.** The CarbonCLIP models are trained for 2 epochs on the MetaCLIP dataset—roughly 40% of the training steps used in prior work (MetaCLIP, TinyCLIP, CLIP). **Despite the one-time cost of optimization, CATransformer achieves overall efficiency gains through reduced training steps post-pruning, along with inference gains that scale with the number of devices.**\n\n[1] Benoit Courty, Victor Schmidt, Sasha Luccioni, Goyal-Kamal, MarionCoutarel, Boris Feld, Jérémy Lecourt, LiamConnell, Amine Saboni, Inimaz, supatomic, Mathilde Léval, Luis Blanche, Alexis Cruveiller, ouminasara, Franklin Zhao, Aditya Joshi, Alexis Bogroff, Hugues de Lavoreille, Niko Laskaris, Edoardo Abati, Douglas Blank, Ziyao Wang, Armin Catovic, Marc Alencon, Michał St˛echły, Christian Bauer, Lucas Otávio N.de Araújo, JPW, and MinervaBooks. mlco2/codecarbon: v2.4.1, May 2024.\n\n**Extended Limitation Discussion**\n\nWe agree that sustainability trade-offs from custom accelerators, such as increased electronic waste and maintenance complexity due to hardware heterogeneity, are important concerns. In the final version, we will discuss potential mitigations, including reusing existing hardware and co-optimizing models with commercially available accelerators.\n\nWe also acknowledge the limitations of proxy-based accuracy estimation. While such proxies enable scalable design space exploration, they can deviate from real-world outcomes due to overfitting to short fine-tuning runs or underestimating pruning’s impact on downstream performance. These approximations may miss interactions between model structure, data, and training dynamics. In future work, we aim to develop more reliable alternatives that improve accuracy estimation without sacrificing scalability."}, {"review_id": "Le8ZyuBFWf", "reviewer": "Reviewer_tjjo", "summary": "The authors introduced CATransformers which is a carbon-aware search framework for sustainability-driven co-optimization of ML models and hardware architectures. They tackled minimizing the total carbon emission including both embodied and operational by jointly optimizing for model's and hardware accelerator's design, and their approach includes three main parts: 1) a multi-objective bayesian optimizer that maximizes accuracy while minimizing latency, energy, and total carbon (embodied + operational) by exploring different model and hardware parameter, 2) ML model evaluator which estimates the accuracy of candidate model architectures during the search, 3) Hardware estimator that provides unified, end-to-end analysis of inference latency and total carbon footprint for each model–hardware configuration. The final output is the sustainable model and hardware configuration.\n\nThe authors evaluated their proposed approach on various models like BERT, Llama, etc. They show their approach can reduce carbon emissions by up to 30% while maintaining accuracy and latency. They also showed that their optimized CLIP model outperformed models like TinyCLIP and baseline CLIP in accuracy and total carbon emissions.", "questions": "1) Can you please clarify what the challenges of using other model archs like CNN could be?\n2) Since there is a trade-off between latency and carbon consumption, do you have an approach to help the user select which configuration tailored to their need?\n3) Is it possible to plug-in CATransformer into PyTorch? what are the challenges?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1) Comprehensive experiments that show their approach can perform in various scenarios\n2) The paper is well-written and organized and informative by having key takeaways for each experiment\n3) The authors also consider embodied carbon as a parameter for total emissions which is a critical parameter\n4) Novel approach to jointly optimize model and hardware arch. from the base when the objective is carbon reduction which tells us the optimization should start from early stages.\n\nWeakness:\n1) The current approach is tailored for edge devices and inference where the carbon optimization in large-scale clusters, e.g., HPC clusters can have greater impact\n2) There is no clear discussion section on what the challenges could be of extending the architectures from transformer-based models to another architecture\n3) While the Pareto frontiers are informative, it could benefit from more intuitive interpretations, e.g., how practitioners should choose between latency and carbon optimization in real deployments remains a bit underspecified.", "was_revised": true, "final_justification": "My questions and concerned were addressed properly.", "rebuttal": "We thank the reviewer for their thoughtful feedback and suggestions. Below, we answer the questions in order: \n\n**Weakness 1: Extensibility to Training and datacenter accelerators**\n\nThis work focuses on making model inference carbon-efficient through hardware-model co-design. Carbon footprint of inference in aggregate is significant due to much wider usages [1]. We also agree with the reviewer that large-scale pretraining of foundation models can result in substantial carbon emissions. As our GPU-based results from Appendices L and M show, our proposed design methodology can scale to datacenter-level accelerators. CATransformer inherently supports training by modeling forward and backward pass latency and energy at the operator level. It can be extended to incorporate parallelization strategies and training system design, enabling optimization of carbon efficiency in distributed training.\n\n[1]  Luccioni, S., Jernite, Y. and Strubell, E., 2024, June. Power hungry processing: Watts driving the cost of ai deployment?. In Proceedings of the 2024 ACM conference on fairness, accountability, and transparency (pp. 85-99). \n\n**Weakness 2, Question1: Generalizability to Non-Transformer architectures**\n\nCATransformer uses Torch.fx to extract operator graphs and estimate latency and energy, making it extensible to any model representable by Torch.fx, including CNNs like ResNet. However, Torch.fx cannot trace dynamic control flow, such as if statements or data-dependent loops, limiting support for models like state-space architectures (e.g., Mamba) that involve input-dependent state transitions. In such cases, the model must be modified or constrained to make the control flow more static for Torch.fx to function properly.\n\nExtending to other model types also requires latency and energy estimations for their unique operators, such as custom kernels or non-standard layers not found in Transformers.  These must first be profiled or estimated for runtime and energy consumption on the target hardware to enable optimization within our framework. If such toolchains become available, CATransformer can integrate them with minimal changes. Additionally, different models may require customized pruning methods. Attention head pruning, for instance, does not apply to CNNs and Mamba-style architectures. \n\nWe will clarify these limitations and generalization challenges in the paper.\n\n**Weakness 3, Question 2: Choosing the optimal configuration** \n\nWe agree that clearer guidance on interpreting the Pareto frontiers would enhance the practical relevance of our work. Choosing an optimal configuration requires balancing carbon emissions and latency based on deployment constraints. For instance, latency-critical applications (e.g., medical imaging, autonomous vehicles) may prioritize low-latency points, while non-critical tasks (e.g., captioning, search) may favor carbon savings. Interactive applications like augmented reality may benefit from a middle ground. Additionally, the trade-off between operational and embodied carbon depends on context. For example, regions with high grid intensity may prefer configurations that reduce operational energy. We will clarify these considerations in the final version.\n\n**Question 3: Challenges in plugging-in CATransformer into PyTorch**\n\nWhile components of CATransformer, such as the optimized Transformer models, can be used within PyTorch due to its open-source foundation and use of HuggingFace models, fully integrating the entire framework into PyTorch is challenging. CATransformer relies on external tools such as CACTI, Accelergy, Phaze, and ACT for detailed hardware and carbon analysis. These capabilities are not supported natively by PyTorch."}, {"review_id": "SZ1m4DQowG", "reviewer": "Reviewer_BWBY", "summary": "The paper introduces CATransformers, a carbon-aware co-optimization framework that jointly searches Transformer architectures and custom accelerator designs to minimize total lifecycle carbon emissions (operational + embodied). The method couples (i) a multi-objective Bayesian optimizer (qNEHVI on Ax/BoTorch) balancing accuracy, latency, energy, and carbon, (ii) a lightweight prune-and-fine-tune evaluator that supplies high-rank-correlated proxy accuracies, and (iii) a hardware estimator that integrates Accelergy, Sunstone, ACT, and Electricity Maps to quantify area, energy, latency, and carbon for candidate accelerators early in design space exploration.\nExperiments span BERT-Base, Llama-3 (8 B), ViT-B/16, and multi-modal CLIP variants. With a 20 TOPS edge budget, carbon-optimized solutions cut total emissions by up to 30 % over latency-optimized and 8 % over energy-optimized baselines, at comparable accuracy. A focused case study yields the CarbonCLIP family, which achieves up to 17 % lower carbon than TinyCLIP while matching or improving accuracy/latency. The framework is to be open-sourced.", "questions": "1. Results assume 1 inference/s for 3 years. How would conclusions change for higher-throughput mobile scenarios (e.g., 10 inf/s) or shorter device lifetimes? A sensitivity plot in the main paper would strengthen applicability.\n2. Appendix K touches on regional grids, but design choices might invert (energy vs. latency) under a zero-carbon grid. Can the optimizer ingest a distribution over grid intensities and return robust designs? Clarify in 3.3/4.5.\n3. You plan to open-source, but key configs (e.g., Cacti SRAM models, yield factors) are unspecified. Please list exact tool versions and parameter files to enable artifact evaluation.\n4. The evaluator fine-tunes each candidate for one epoch on MS-COCO/GLUE etc. What is the wall-clock overhead per candidate, and is it amortized across search iterations? A brief cost breakdown would help readers gauge scalability.\n5. Carbon is central, but other sustainability axes (e-waste, water, mineral use) may correlate with chip area. Could CATransformers be extended to multi-impact objectives? A short discussion in the conclusion would increase societal relevance.\n\nScores could rise with clearer reporting of (1)–(3) and an expanded robustness study for different duty cycles / grids.", "limitations": "Authors acknowledge lack of GPU/CNN support, focus on edge inference, and fixed carbon-intensity/lifetime assumptions. Suggested to foreground these in  Sec 5 and add an explicit paragraph on potential negative societal impacts beyond carbon (privacy, resource inequity).", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Introduces a novel, holistic framework that co-optimizes both ML models and hardware accelerators to directly minimize total lifecycle carbon emissions (operational + embodied). Novel combination of Bayesian NAS with joint embodied + operational carbon objective; extends prior work (e.g., CE-NAS) from model-only to full stack. Multi-modal CLIP optimisation under carbon constraints is new.\n2. First study to treat carbon as first-class search objective across model and hardware; could influence sustainable ML practice on edge devices. CarbonCLIP models are likely reusable by practitioners focusing on low-footprint deployments.\n2. The authors cross-check latency/energy estimates against real GPUs (≤ 9 % error), show a Spearman 0.98 correlation between proxy- and full-accuracy, and report low hyper-volume variance (< 3.5 %) across search runs, evidence that both the estimator and optimizer are reliable.\n3. Empirical results across diverse architectures (BERT, Llama-3, ViT, CLIP) demonstrate up to 30 % emission cuts over latency-optimized baselines, with the CarbonCLIP family delivering ~17 % lower carbon while maintaining accuracy.\n4. Clear visualizations (Pareto fronts, ISO-accuracy plots) effectively communicate trade-offs and design insights.\n\nWeaknesses:\n1. Restrictive carbon assumptions i.e. fixed 1 inf/s duty cycle, 3-year lifetime, California grid intensity, and Taiwan 22 nm fabrication limit generalizability; alternative scenarios are only briefly treated in the appendix.\n2. estimator details (e.g., specific Accelergy/Sunstone knobs, ACT parameters) are not fully documented, which may hinder independent reproduction\n3. targets edge-inference accelerators; training-time carbon and data-center/cloud inference remain unaddressed.\n4. several tables are hard to parse, some abbreviations go undefined, and embodied-carbon scaling equations (fundamental to the paper) are omitted from the main text.\n5. There are several works [1-5] that try to answer sustainability related issues in AI as well as estimate CO2 footprint. I think listing them would benefit the paper and give a more holistic understanding of how researchers have been trying to tackle this problem before.\n\n[1] Bouza, L., Bugeau, A. and Lannelongue, L., 2023. How to estimate carbon footprint when training deep learning models? A guide and review. Environmental Research Communications, 5(11), p.115014.\n[2] Luccioni, S., Jernite, Y. and Strubell, E., 2024, June. Power hungry processing: Watts driving the cost of ai deployment?. In Proceedings of the 2024 ACM conference on fairness, accountability, and transparency (pp. 85-99).\n[3] Luccioni, S., Gamazaychikov, B., Hooker, S., Pierrard, R., Strubell, E., Jernite, Y. and Wu, C.J., 2024. Light bulbs have energy ratings—so why can’t AI chatbots?. Nature, 632(8026), pp.736-738.\n[4] Gowda, S.N., Hao, X., Li, G., Gowda, S.N., Jin, X. and Sevilla-Lara, L., 2025. Watt for what: Rethinking deep learning’s energy-performance relationship. In European Conference on Computer Vision (pp. 388-405). Springer, Cham.\n[5] Strubell, E., Ganesh, A. and McCallum, A., 2020, April. Energy and policy considerations for modern deep learning research. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 09, pp. 13693-13696).", "was_revised": true, "final_justification": "The authors have responded to my queries and I have no significant concern remaining.", "rebuttal": "We thank the reviewer for the detailed review of our work and thoughtful suggestions. Below, we answer the questions and comments in order. \n\n\n**Question 1, Weakness 1: Tuning lifetime and regional parameters**\n\nWe fully agree that factors such as grid carbon intensity, device lifetime, and inference frequency significantly impact the total carbon footprint. **To support diverse deployment scenarios, CATransformer exposes all of these parameters as configurable inputs.** In addition, hardware design parameters, such as process technology, are also fully configurable. \n\nThese factors primarily influence the balance between operational and embodied carbon. For example, high inference frequency, long device lifetime, or deployment in regions with high grid carbon intensity increases operational energy consumption, making operational carbon the dominant contributor. Conversely, in scenarios with low inference frequency, short device lifetime, or low grid carbon intensity, embodied carbon plays a more significant role.\n\nWe include a sensitivity analysis on inference frequency. The table below reports the percentage of operational carbon as a fraction of total carbon footprint across CarbonCLIP model variants. Increasing the inference rate from 1 to 10 inferences per second leads to an average of 50% increase in the operational carbon share.\n| Model         | Ratio of Operational Carbon (%)|   |      |             |\n|---------------|:-------------------------------:|:-----------:|:-----------:|:-----------:|\n|               | 1 inference / sec                       | 3 inference/ sec | 5 inference/ sec | 10 inference/ sec |\n| CarbonClip-XS | 18                              | 40          | 53          | 69          |\n| CarbonClip-S  | 27                              | 52          | 65          | 78          |\n| CarbonClip-M  | 32                              | 59          | 71          | 83          |\n| CarbonClip-L  | 36                              | 63          | 74          | 85          |\n| CarbonClip-XL | 44                              | 71          | 80          | 89          |\n\nAppendix K complements this analysis by demonstrating how external factors like grid carbon intensity influence whether CATransformer prioritizes reducing operational energy or minimizing hardware footprint. **Importantly, the trends observed with varying grid intensity generalize to changes in inference frequency and device lifetime, as all induce similar shifts in the operational versus embodied carbon tradeoff.** This confirms that CATransformer adapts its optimization strategy consistently across deployment scenarios—whether in high-throughput mobile environments or in systems with shorter lifespans.\n\nWe will clarify these insights in the final version of the paper to better highlight CATransformer’s flexibility and its nuanced handling of carbon-aware optimization.\n\n**Question 2: Carbon zero regions and varying grid intensities**\n\nWe thank the reviewer for important questions regarding carbon assumptions. On a zero-carbon grid, operational carbon becomes negligible, shifting optimization entirely to minimizing embodied carbon, potentially favoring smaller hardware. In such cases, the pruned submodel size and architecture are likely determined solely by latency constraints, since operational energy (i.e., the energy required to run the model) is no longer a factor. The goal then becomes to run the largest possible model (to maximize accuracy) on the smallest hardware that still meets the latency requirement. While CATransformer currently uses a single grid intensity, it can be readily extended to ingest a distribution of intensities, enabling robust designs for varying regional or fluctuating grid conditions. These clarifications will be added to sections 3.3 and 4.5.\n\n**Question 3, Weakness 2: Details of estimators and open source tools used**\n\nWe agree with the reviewer that providing exact parameters and configurations for each toolchain is essential for reproducibility. We have submitted the full codebase as supplementary material, including all configuration files, tool versions, and experimental parameters. These details will also be made publicly available upon open-sourcing the code. Below is a summary of the key tools and configurations used:\n* CACTI: Version 7, using 22nm process technology node configurations from commit 1ffd8d\n* Accelergy: Version 0.4\n* ACT: Used an IC yield factor of 0.875 and accounts for the carbon footprint of manufacturing gases per unit area, assuming 95% abatement.\n* Carbon Intensity: Based on 2024 data from Electricity Maps. Exact values are specified in the configuration files:\n  * California: 224 CO2eq/kWh\n  * Taiwan: 524 CO2eq/kWh\nWe will ensure that all configuration files and tool references are clearly documented and accessible in the final open-source release to support artifact evaluation and reproducibility.\n\n**Question 4: Cost of optimization process**\n\nRuntime per candidate varies with model size and available compute. In our experiments, optimization was conducted on a single node with 8 V100-SXM2-16GB GPUs. Under this setup, the full optimization takes ~5 hours for BERTbase and up to 20 hours for CLIP, each with 100 trials. This translates to 3–12 minutes per trial, including ~2 minutes for hardware estimations. Fine-tuning adds another 1–10 minutes per candidate, depending on model size. With newer or more powerful hardware, per-trial costs would decrease proportionally. To reduce overhead, we implement result caching, avoiding redundant retraining of previously evaluated candidates across hardware settings and amortizing cost over search iterations.\n\n**Question 5: Extension to other sustainability axes**\n\nWhile CATransformer currently focuses on carbon footprint, we agree that sustainability extends to factors like electronic waste, water usage, and rare mineral consumption. Our optimizations, which often reduce chip area, can implicitly benefit these metrics. In principle, CATransformer could support multi-objective optimization over broader environmental impacts. The main challenge lies in the current lack of standardized, fine-grained data for these additional sustainability metrics. We will highlight these considerations in the conclusion to underscore the broader societal relevance and need for holistic, sustainable hardware design.\n\n**Weakness 3: Extensibility to training and datacenter accelerators**\n\nThis work focuses on making model inference carbon-efficient through hardware-model co-design. Carbon footprint of inference in aggregate is significant due to much wider usages [1]. We also agree with the reviewer that large-scale pretraining of foundation models can result in substantial carbon emissions. As our GPU-based results from Appendices L and M show, our proposed design methodology can scale to datacenter-level accelerators. CATransformer inherently supports training by modeling forward and backward pass latency and energy at the operator level. It can be extended to incorporate parallelization strategies and training system design, enabling optimization of carbon efficiency in distributed training.\n\n[1]  Luccioni, S., Jernite, Y. and Strubell, E., 2024, June. Power hungry processing: Watts driving the cost of ai deployment?. In Proceedings of the 2024 ACM conference on fairness, accountability, and transparency (pp. 85-99). \n\n\n**Weakness 4: Improvements on formatting for Tables and Abbreviations.**\n\nIn the final version of the paper, we will enhance the presentation of tables and ensure that all abbreviations and equations are clearly defined.\n\n**Weakness 5: Related works in Sustainability in AI**\n\nWe thank the reviewer for highlighting these relevant works on AI sustainability. Broadly, they address the topic through three main approaches: (1) Quantifying the carbon footprint of training: [1] and [5] estimate the emissions and energy use of training deep learning models or evaluate tools for doing so;  (2) Analyzing Deployment Energy Use and emissions: [2] and [3] focus on energy consumption during inference and advocate for standardized environmental assessment frameworks; and (3) Exploring Energy-Performance Trade-offs: [4] explores the balance between model accuracy and electricity use, proposing metrics to encourage more energy-efficient architectures. We will incorporate a discussion of these works in Section 2 of the final version.\n\n\n**Limitations**\n\nWe agree that broader societal impacts, such as privacy concerns in edge deployment and potential resource inequities from hardware specialization, are important considerations. We will include a discussion of these challenges, alongside the limitations of fixed carbon-intensity and lifetime assumptions, to provide a more balanced perspective on the societal implications of our work."}]

Author rebuttal:

Meta-review: The authors introduce CATransformers,  a unified optimization pipeline for carbon-efficient Transformer-based models and hardware accelerators.  The approach uses a base  pre-trained Transformer to generate pruned variants and a hardware template.  The algorithm uses a lightweight fine-tuning of pruned models to ensure satisfactory accuracy as untrained pruned models provide poor performance.   Since the optimization problem cannot be optimized easily, CATransformers uses multi-objective Bayesian optimization  to maximize accuracy while minimizing in particular total carbon emissions. With this contribution, the authors proposed an end-to-end analysis of  total carbon footprint for the explored models and hardware configurations.

Reviewers were convinced by several key features of the proposed approach in particular the use of carbon emissions as a primary optimization objective for Transformer-based models to foster sustainable ML practice,  the extensive experimental analysis of model objectives,  scales, and hardware configurations. The experiments are well described and demonstrate significant emission cuts in various frameworks.
 
Key details were provided during the discussion with reviewers, in particular on the supplementary material to ensure reproducibility. The authors also clarified all the parameters that can be used as configurable inputs to support deployment in many settings. The authors are encouraged to clarify the generalizability to architectures which are not transformer-based to detail the challenges to extend the optimization to other model. They should also discuss possible limitations with respect to global deployment of sustainable ML models. 

Overall, the rebuttal highlighted that the paper has a great potential and can be of interest for a large ML audience.
Decision: Accept (poster)