Title: CATransformers: Carbon Aware Transformers Through Joint Model-Hardware Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Carbon Footprint

Sustainability

Hardware Accelerator

Architecture Search

Neural Architecture Search

Optimization
Abstract: Machine learning solutions are rapidly adopted to enable a variety of key use cases, from conversational AI assistants to scientific discovery. As the adoption of machine learning models becomes increasingly prevalent, the associated lifecycle carbon footprint is expected to increase, including both *operational carbon* from training and inference and *embodied carbon* from AI hardware manufacturing. We introduce CATransformers, the first carbon-aware co-optimization framework for Transformer-based models and hardware accelerators. By integrating both operational and embodied carbon into early-stage design space exploration, CATransformers enables sustainability-driven model architecture and hardware accelerator co-design that reveals fundamentally different trade-offs than latency- or energy-centric approaches. Evaluated across a range of Transformer models, CATransformers consistently demonstrates the potential to reduce total carbon emissions --by up to 30\% -- while maintaining accuracy and latency. We further highlight its extensibility through a focused case study on multi-modal models. Our results emphasize the need for holistic optimization methods that prioritize carbon efficiency without compromising model capability and execution time performance. The source code of CATransformers is available at https://github.com/facebookresearch/CATransformers.

CATransformers: Carbon Aware Transformers Through Joint Model-Hardware Optimization

Abstract

Machine learning solutions are rapidly adopted to enable a variety of key use cases, from conversational AI assistants to scientific discovery. This growing adoption is expected to increase the associated lifecycle carbon footprint, including both operational carbon from training and inference and embodied carbon from AI hardware manufacturing. We introduce CATransformers-the first carbonaware co-optimization framework for Transformer-based models and hardware accelerators. By integrating both operational and embodied carbon into early-stage design space exploration, CATransformers enables sustainability-driven model architecture and hardware accelerator co-design that reveals fundamentally different trade-offs than latency- or energy-centric approaches. Evaluated across a range of Transformer models, CATransformers consistently demonstrates the potential to reduce total carbon emissions-by up to 30%-while maintaining accuracy and latency. We further highlight its extensibility through a focused case study on multi-modal models. Our results emphasize the need for holistic optimization methods that prioritize carbon efficiency without compromising model capability and execution time performance. The source code of CATransformers is available at https://github.com/facebookresearch/CATransformers .

1 Introduction

As machine learning (ML) systems become more widespread across various industries, it is crucial to take a closer examination of their carbon footprint and find strategies to mitigate it across the system stack. Sustainable ML system design requires a holistic approach that considers both operational carbon (energy used during training and inference) and embodied carbon (emissions from hardware manufacturing and lifecycle) [GEH + 22], which together contribute to total emissions. This work tackles the question: How does incorporating carbon footprint metrics into optimization workflows influence the design of ML models and hardware architectures as a co-optimization?

A fundamental challenge in designing sustainable AI systems stem from the tight coupling between model architecture and hardware design, which together shape both operational and embodied carbon. Model execution on a particular hardware determines runtime characteristics, such as compute intensity and memory access patterns, which affect operational carbon, while hardware parameters like chip area, memory hierarchy, and fabrication technology drive embodied carbon. Carbon optimization requires joint model-hardware exploration, particularly during early accelerator design, to uncover opportunities that align model demands with hardware capabilities in a carbon-efficient way. Co-design is even more important for multi-modal workloads like vision-language models. Each modality introduces distinct bottlenecks-e.g., vision transformers usually have compute-bound workloads [DGC24], while text transformers have more memory-bound workloads [Fu24]. Co-

TinyCLIP-8M

TinyCLIP-39M

CarbonCLIP-XS

CarbonCLIP-S

8% better accuracy; 3% less carbon

<10ms latency

Same accuracy; 17% less carbon

<15ms latency

60

50

40

R

30

20

0.30

0.35

0.40

0.45

Carbon (kg CO2e)

optimization across modalities and system layers is essential to minimize the total carbon footprint without compromising performance.

While this sustainability challenge applies across the entire AI lifecycle, the rapid adoption of AI on edge devices presents a particularly acute and fast-growing problem [QPFM + 23, GKL + 22]. With billions of products like smartphones and AR/VR headsets requiring specialized, custom accelerators [WLS + 24, SWL + 22, HAI25, Goo20, Nvi25], the embodied carbon from manufacturing constitutes a massive, upfront environmental cost. Furthermore, the cumulative operational carbon from continuous inference over a device's multi-year lifespan represents a substantial and often overlooked energy demand [App21]. Addressing the tightly coupled model-hardware system at the edge is therefore a critical first step, offering a high-impact domain to develop and validate the carbon-aware co-design principles needed for sustainable AI.

Prior hardware-aware neural architecture search (NAS) methods have focused primarily on latency or energy and are not suitable for total carbon optimization. This is because carbon is a complex metric that cannot be captured by optimizing latency or energy in isolation. These approaches either neglect embodied carbon by optimizing only the model on fixed hardware [WWL + 20] or treat latency, energy, and chip area as orthogonal constraints [ZDM + 22], overlooking the tightly coupled nature of operational (latency- and energy-driven) and embodied (chip area-driven) carbon. As a result, these methods are fundamentally limited in their ability to minimize total carbon emissions. Furthermore, prior work often assumes fixed hardware or fixed model backbones, limiting their applicability to a broader range of emerging hardware platforms and modern, multi-modal models.

We introduce CATransformersthe first framework to jointly optimize model and hardware architecture with the specific goal of minimizing total carbon emissions for edge inference-only devices. Distinct from prior latency- or energy-centric methods, our approach incorporates both operational and embodied carbon into a unified optimization pipeline , revealing carbon-efficient configurations previously overlooked. We observe that optimizing traditional metrics in isolation frequently results in suboptimal trade-offs. For example, accelerating inference by increasing compute density may inadvertently increase power consumption and embodied carbon due to larger chip area or more resource-intensive fabrication. By prioritizing total carbon as a first-class objective, CATransformers yields fundamentally different and more sustainable design choices.

CATransformers systematically explores the joint design space of model architectures and hardware accelerators using multi-objective optimization. It comprises three main components: (1) a MultiObjective Bayesian Optimizer that balances accuracy, latency, energy, and carbon emissions; (2) an ML Model Evaluator that efficiently navigates model variants using importance-based pruning and fine-tuning; and (3) a Hardware Estimator that profiles latency, energy, and carbon footprint.

We evaluate CATransformers on language, vision, and multi-modal Transformer-based models and show that its co-optimized model-hardware configurations reduce total carbon by 30% over latencyoptimized and 8% over energy-optimized baselines. To further highlight its utility, we present CarbonCLIP, a family of CATransformers-optimized CLIP models.

Figure 1 shows CarbonCLIP achieves up to 17% lower total carbon emissions compared to stateof-the-art CLIP variants on edge devices, while maintaining similar accuracy and latency. Joint

optimization enables low-carbon, high-accuracy designs that outperform approaches that optimize only hardware configurations.

Our key contributions include:

Insights and Analysis: Empirical insights showing how carbon-aware optimization shifts model-hardware trade-offs relative to traditional metrics, enabling early-stage design exploration for next-generation ML accelerators.

Quantification Framework : We develop the first open-source toolchain to estimate both operational and embodied carbon for custom accelerators during design-space exploration.

Carbon-Aware Co-optimization: CATransformers uses multi-objective Bayesian optimization to jointly explore model and hardware design spaces, balancing accuracy, latency, energy, and carbon. It leverages fine-tuning-based proxy signals to reduce evaluation cost.

Sustainable Multi-Modal Models : Using CATransformers, we co-optimize CLIP variants (CarbonCLIP) that reduce total carbon by up to 17% compared to edge-deployed CLIP baselines, without sacrificing accuracy or latency.

CATransformers is designed as a modular, extensible framework that supports the seamless integration of diverse model architectures, optimization strategies, and carbon estimation techniques. The goal is not to outperform other NAS frameworks on traditional metrics, but to highlight the fundamentally different design trade-offs that emerge under carbon-centric optimization objectives. Through systematic comparisons across multiple optimization modes and model families, we demonstrate the practical impact of carbon-aware design, which is the core contribution of this work.

2 Background and Related Works

Hardware Accelerator Search: Specialized hardware accelerators have been developed to efficiently run deep learning workloads, with tensor cores for matrix operations [JKL + 23, MPA + 16, PSM + 17, CKES16] and vector cores for element-wise operations [JYP + 17, GKX + 24]. Prior accelerator search frameworks [APK + 24, PMK + 22, WTPM24, PMT25, ZHS + 22, SSL23] optimize for throughput and energy, but do not jointly optimize hardware and model architectures or consider carbon footprint in the design process.

Hardware and Neural Architecture Co-optimization: Co-optimization methods [ZDM + 22, CHY + 21, JYS + 20, LYH21] typically focus on latency or energy, ignoring carbon as a primary design objective. Approaches that do consider carbon [ECA + 23, ECA + 25, ZG23] often optimize either the model or hardware in isolation. Furthermore, prior work often assumes fixed hardware or fixed model backbones, limiting applicability to a broader range of emerging hardware platforms and modern, heterogeneous models-particularly multi-modal architectures. Our work addresses this gap by jointly optimizing both hardware and model architectures with total carbon footprint-including embodied and operational emissions-as a first-class objective.

The Carbon Footprint of AI Systems: Much of the research on the carbon footprint of AI focuses on operational emissions, with three main directions: (1) quantifying the emissions from training [BBL23, SGM20, LLSD19, WRG + 22], (2) analyzing energy use during deployment and inference [LJS24, LGH + 24], and (3) developing techniques to reduce emissions while exploring energy-performance trade-offs [GHL + 25, ZLJG24, LJGT24, ALK + 23].

Embodied carbon, however, remains underexplored. Tools like ACT [GEH + 22], IMEC.netzero [IME25], and LLMCarbon [FKW + 24] have begun to estimate embodied emissions, but a holistic framework that quantifies and minimizes both operational and embodied carbon remains lacking. CORDOBA [ECA + 25] introduces a carbon-aware accelerator design tool, but it is not open-sourced, limiting reproducibility and extensibility. Unlike CE-NAS [ZLJG24], which reduces carbon emissions during neural architecture search, CATransformers is the first to optimize total carbon emissions from both model and hardware, during early-stage design.

AI Systems on the Edge: AI models are no longer confined to large-scale datacenters and are now widely deployed on edge devices like smartphones, AR/VR headsets, robotics, and vehicles [QPFM + 23, SWL + 22, HAI25]. These edge settings present unique challenges, including strict resource constraints, hardware heterogeneity, and communication bottlenecks that impact

HW

Architecture

Template

Multi-Objective Bayesian Optimizer

ML Model Evaluator

HW Estimator

Proxy

Latency,

Energy,

Carbon

Metrics and

Constraints

Output

Submodel &

Submodel

performance [WNM23, HYW + 23, bJRE + 23]. This wide adoption, however, also raises significant environmental concerns. The sheer scale of device production means embodied carbon from chip fabrication is a major contributor to total emissions [GKL + 22]. Simultaneously, the cumulative operational carbon from frequent inference over a device's multi-year lifespan presents a distinct and growing sustainability challenge. CATransformers aims to address this gap by enabling holistic, carbon-aware co-design of AI models and hardware accelerators specifically for edge systems.

CLIP Models and Edge Variants: Multi-modal models like CLIP [RKH + 21] combine Transformerbased text encoders with image encoders (ResNet [HZRS16] or ViT [DBK + 21]) and are trained on large datasets [SVB + 21, XXT + 24, SBV + 22, GIF + 23] to learn cross-modal associations. These models support tasks like zero-shot classification and retrieval. Several variants [WPZ + 23, LBL + 24, STJ + 23, VPF + 24] adapt CLIP for edge devices through pruning and efficient training, but focus solely on accuracy and latency. Our work is the first to optimize CLIP models for edge deployment with total carbon emissions in mind, without sacrificing accuracy or latency.

3 Framework Overview

In this section, we introduce CATransformers, a carbon-aware architecture search framework for sustainability-driven co-optimization of ML models and hardware architectures. As illustrated in Figure 2, CATransformers framework takes three inputs: (1) a base ML model, (2) a hardware architecture template, and (3) a set of optimization objectives and constraints that define the joint model-hardware search space. The framework consists of three key components: a multi-objective optimizer, an ML model evaluator, and a hardware estimator.

3.1 CATransformers Inputs

Base Model: The base model is a large, pre-trained Transformer that serves as the foundation for generating pruned variants. It defines the architecture's shape, functionality, and search space. As shown in Figure 3a, pruning is applied along several dimensions, including the number of layers, hidden size, feedforward network width, attention heads, and embedding dimensions.

Hardware Architecture Template: The accelerator template (Figure 3b) captures key components and tunable parameters inspired by academic and industry designs [JYP + 17, APK + 24, ZHS + 22, WTPM24]. It features tensor cores with Processing Elements (PEs) arranged in X and Y dimensions for GEMM operations, vector units for element-wise operations, local buffers for data reuse, and a shared global SRAM and off-chip memory-consistent with existing edge inference accelerators [SWL + 22, WLS + 24]. Increasing PEs or cores boosts performance but raises area and energy costs; more on-chip memory lowers latency but adds energy overhead. The optimal hardware configuration depends on the model architecture, size, and performance constraints. Table 1 summarizes the hardware design space parameters.

3.2 MLModel Evaluator

The ML model evaluator estimates the accuracy of candidate model architectures during the search.

Text / Vision Encoder

Layer n

Layer 2

…

Head

Head

Head

Head

…

Layer n-1

Embedding

FFN 2

FFN 1

Pruned

blocks:

Layer 1

(a) Overview of the pruning dimensions

Core

Core

Core

Core

Off-chip DRAM

Vector Unit

GEMM Unit

PE_x

PE_y

…

…

…

…

(b) Architecture template

Pruning : The evaluator prunes the pre-trained base model along key dimensions, including the number of layers, feedforward network size, attention heads, and embedding dimension. For multimodal models like CLIP, each Transformer (text and vision) is pruned independently using strategies from prior work [LBL + 24, SDDN20, WWL + 20]. Each layer is pruned uniformly. See Appendix A for pruning details and empirical observations.

Fine-tuning for Accuracy Proxy : Direct evaluation of untrained pruned models results in poor accuracy, while fully retraining every model is computationally infeasible. Instead, we use lightweight fine-tuning to approximate accuracy. Our ablation studies show that this method yields a high Spearman correlation (0.98) with fully trained accuracy, making it a reliable proxy for ranking candidate models (Appendix C). This approach allows accurate and efficient evaluation within the optimization loop, enabling scalable exploration of Transformer-based models without full retraining. For language models we use the MRPC task from the GLUE dataset [WSM + 18] for semantic similarity detection, for vision models, the CIFAR-10 dataset [Kri09] for image classification; and for CLIP models, the MSCOCO dataset [LMB + 15] for retrieval.

3.3 Hardware Estimator

The hardware estimator provides unified, end-to-end analysis of inference latency and total carbon footprint-including both embodied and operational emissions-for each model-hardware configuration. It leverages performance and energy models to estimate operator-level latency, energy, and area [WES19, ONFL23], and incorporates carbon modeling for manufacturing-related emissions and location-specific operational carbon, scaled over the system's deployment lifetime [GEH + 22, Map25]. To the best of our knowledge, this represents the first open-source toolchain built for early-stage design space exploration that holistically quantifies latency, energy, and total carbon emissions of custom ML accelerators. A detailed workflow is provided in Appendix B.

3.4 Multi-Objective Optimization

CATransformers uses multi-objective Bayesian optimization to efficiently explore the joint design space of model architectures and hardware configurations building on Ax [MP24] and

BoTorch [BKJ + 20] with the qNEHVI algorithm [DBB21]. Compared to reinforcement learning or evolutionary search, Bayesian optimization offers better sample efficiency and uncertainty modeling, making it ideal for our static, high-dimensional search space where each configuration takes 5-15 minutes to evaluate. While prior work [WWL + 20] explored just 125 models over 30 iterations, CATransformers scales to a search space of 100 million configurations in only 100 iterations.

The optimization maximizes accuracy while minimizing latency, energy, and total carbon (embodied + operational), exploring model (Figure 3a) and hardware parameters (Table 1) under a TOPS-based compute budget based on publicly available edge accelerators. Our framework supports four computeconstrained modes: (1) Accuracy & Total Carbon (with latency constraint), (2) Accuracy & Latency, (3) Accuracy & Energy, and (4) Accuracy, Latency, & Total Carbon.

3.5 Outputs

CATransformers outputs optimized combinations of model and hardware configurations that, when deployed together, improve overall efficiency. Once the optimizer identifies carbon-efficient, pruned model architectures, these models can optionally be fine-tuned to recover any accuracy loss. Preliminary results show these pruned models require far fewer training steps than full pre-training. Specifically, CarbonCLIP models are fine-tuned for just 2 epochs on MetaCLIP [XXT + 24], using only 40% of the training steps compared to prior works [RKH + 21, XXT + 24, WPZ + 23].

4 Evaluation

4.1 Experimental Settings

Model: To demonstrate the versatility of CATransformers, we apply it to diverse transformerbased architectures: encoder-only Bert-Base [DCLT19], decoder-only Llama3-8B [GDJ + 24], vision transformer ViT-B/16 [DBK + 21], and multi-modal CLIP models (CLIP-ViT-B/16 and CLIP-ViTB/32)[RKH + 21]. We use pre-trained weights from HuggingFace[WDS + 19] for Bert, Llama, and ViT, and adopt OpenCLIP's [IWW + 21] DataComp-1B [GIF + 23] models for CLIP. Building on CLIP optimization results, we generate CarbonCLIP models with corresponding hardware designs.

Baselines: Prior work primarily targets performance or energy efficiency, rarely considering carbonaware hardware-model co-design. These approaches are typically evaluated on fixed hardware and model architectures, making direct comparison difficult. To ensure a fair and meaningful evaluation, we benchmark CarbonCLIP against two widely adopted baselines: (1) standard CLIP models (ViTB/16, ViT-B/32), and (2) TinyCLIP, a state-of-the-art variant optimized for latency. For each, we apply CATransformers's hardware search with fixed model architectures to derive comparable accelerator designs. This allows us to isolate the benefits of carbon-aware co-optimization. Our results show that carbon-optimized models can match or exceed the accuracy and latency of traditional baselines-demonstrating the practical value of our approach.

Hardware Estimation and Validation: We estimate accelerator area, latency, and energy using open-source tools (Appendix B), assuming 22nm process technology and 8-bit integer operations. To validate latency estimates, we use SCALE-Sim [SZW + 18, SJZ + 20] for CLIP's QKV projections. Our estimates have an average latency error of 13%, consistent with prior work [PRS + 19, WES19], and are validated on standard compute array sizes (16×16 to 64×64). We further validate energy and latency predictions against real GPU hardware (V100, A100, H100) by modeling GPU-like architectures. Our estimates closely match measured results, with average errors of 8% (energy) and 9% (latency), and Spearman's rank-order correlation r ranging from 0.5 to 1.0 and p < 0 . 05 , confirming toolchain accuracy. Details of the validation is in Appendix M.

Carbon Emission Estimation: Operational carbon is estimated using California grid intensity over a 3-year lifespan (1 inference/sec, 6 hrs/day), reflecting typical mobile usage [App21, NSY24, Har25]. Embodied carbon is calculated assuming fabrication in Taiwan. Appendix K explores the impact of different deployment regions, energy sources, and optimization metrics.

Execution Setup: Bayesian optimization is performed on a single node (8×V100 GPUs, 80 CPUs) over 100 trials, taking 5-20 hours depending on the model. Post-pruning training of CarbonCLIP uses 224 GPUs on the MetaCLIP-2.5B dataset [XXT + 24], with a batch size of 128, learning rate of

Carbon

Carbon+Latency

Energy

Latency

15

8

40

40

sw)

S

10

CLIP-

ViT-B16

6

BERT

30

Llama3

30

ViT-B16

Latency

4

20

20

2

10

10

0

0

0

0

0.48

0.50

0.53

0.55

0.28

0.30

0.32

0.34

0.36

0.30

0.33

0.35

0.38

0.40

0.330.350.380.400.430.450.48

Carbon(kgCO2e)

5 × 10 -4 , and 2 distillation epochs. We quantify the Carbon footprint of running CATransformers in Appendix N.

4.2 Joint Model and Hardware Architecture Search Using Different Metrics

In this section, we use CATransformers to perform joint model-hardware architecture search, with carbon footprint as a central design metric alongside traditional objectives like accuracy, latency, and energy. We evaluate different optimization modes under a 20 TOPS compute budget, representative of modern edge accelerators [HAI25, Nvi25]. When latency is not an optimization target, a maximum latency constraint of 50ms is enforced [vMVJ22] to ensure realistic specifications. Each experiment is repeated three times, and we compute the Hypervolume (HV) indicator to assess consistency. HV measures the portion of the objective space dominated by the Pareto front relative to a reference point. Across runs, its standard deviation is below 0.03, with an average coefficient of variation under 3.5%, indicating statistically consistent results. Appendix F provides a detailed breakdown of the statistics.

Takeaway 1: Carbon optimization yields the lowest footprint but at the cost of latency; energy optimization strikes a better balance.

Figure 4 shows the latency-carbon trade-off at iso-accuracy points (based on proxy accuracy before fine-tuning). Full configurations across additional accuracy levels for each model are detailed in Appendix D. Comparing optimization modes, we find that carbon optimization significantly reduces carbon footprint by an average of 30%, but with a 7.7 × increase in latency compared to latencyoptimized baselines. Energy-optimized configurations reduce carbon by 24% while limiting latency overhead to 4 × . Joint optimization for Carbon and Latency achieves a 18% carbon reduction with minimal latency increase, effectively balancing sustainability and performance.

Takeaway 2: Energy optimization indirectly reduces latency, but not as effective as direct latency optimization.

Minimizing energy tends to reduce both power and computation time, often lowering latency. In contrast, carbon-focused optimization emphasizes minimizing hardware area to reduce embodied carbon, often resulting in slower, smaller designs. Because total energy consumption is proportional to latency (i.e., energy = power × delay), latency cannot be extended arbitrarily when using smaller-area hardware. Excessively long latencies not only fail to meet practical performance targets but also begin to significantly increase operational carbon costs. Still, energy optimization does not always guarantee low latency. For example, Bert and Llama3 configurations in Figure 4 show higher latency despite energy-focused design. Direct latency optimization, seen in Latency-only and Carbon+Latency modes, consistently delivers the lowest-latency results across all models. These trade-offs highlight the need to carefully balance carbon constraints and real-world performance demands.

Takeaway 3: Latency-optimized designs favor large, high-throughput hardware; carbon-optimized designs use compact, low-power accelerators.

Latency-focused designs often use up to 3 × larger area, enabled by more compute units and larger memory hierarchies. These systems frequently select larger models that fully utilize hardware to meet latency targets. In contrast, energy-optimized designs typically pair smaller models with larger accelerators to minimize delay and energy per inference. Carbon-optimized designs prioritize smaller hardware to reduce embodied carbon, at the cost of performance. For example, on BERT-Base (Figure 4), the carbon-optimized system uses a single-core accelerator with 512 Processing Elements (PEs), 64KB local memory, and 2MB global memory. The latency-optimized design uses two cores,

each with 4K PEs, 128KB local memory, and 4MB global memory-cutting latency by over 4 × but increasing carbon footprint by 26%.

Takeaway 4: Model sensitivity to pruning varies by model architecture: CLIP and ViT are especially sensitive to hidden dimension pruning.

Each model architecture responds differently to pruning. CLIP and ViT models are particularly sensitive to reductions in the hidden dimension, leading CATransformers to typically follow this pruning order for these models: FFN dimension → attention heads → layers → embedding dimension. This sensitivity arises because ViTs split images into relatively few patch tokens, requiring each token to carry rich semantic and spatial information. The hidden dimension governs the expressiveness of these token embeddings, and pruning it severely limits ViT's ability to model visual content-especially since ViTs lack the hierarchical features of CNNs and rely on global attention.

CLIP models are especially sensitive due to their multi-modal nature, where precise alignment between visual and textual embeddings is crucial. Pruning hidden or embedding dimensions can significantly degrade performance (Appendix A), as excessive pruning of the vision encoder disrupts alignment with the text encoder, harming model accuracy. Additionally, the text encoder is often pruned more aggressively than the vision encoder, which is essential for processing complex visual inputs. In contrast, language models such as Bert and Llama3 tend to be more robust to pruning in the hidden dimension. These observations underscore the challenges of pruning multi-modal models like CLIP and motivate our focus on their optimization throughout the paper.

4.3 Hardware Optimization for Fixed CLIP Models

While the previous section highlighted CATransformers' general utility, we now focus on CLIP-a widely used multi-modal model combining text and vision encoders. CLIP poses additional challenges due to its heterogeneous computation and the complex interplay between modality-specific pruning and system performance. We first evaluate the carbon footprint of state-of-the-art CLIP variants. To ensure fair comparison, we use CATransformers' hardware search to optimize inference latency and carbon footprint under a fixed model and a 20 TOPS compute budget. This yields a Pareto frontier illustrating optimal latency-carbon trade-offs. Table 2 summarizes configurations with minimum carbon and minimum latency for each model. As with joint optimization, carbon-focused hardware design results in smaller accelerators with fewer compute and memory resources. While this reduces area and emissions, it typically increases inference latency.

Takeaway 5: Hardware must be tailored to the model-parameter count, patch size, and modalityspecific traits affect optimal configurations.

Results from the fixed-model hardware search underscore the importance of hardware-model codesign: optimal performance and carbon efficiency require aligning hardware with model architecture. ViT patch size notably influences accelerator dimensions. For instance, TinyCLIP-61M/32-despite having more parameters-outperforms TinyCLIP-39M/16 in both latency and carbon footprint when paired with hardware better suited to its larger patch size. ViT-B/32 produces a sequence length of 50, compared to 197 for ViT-B/16, enabling more efficient execution on smaller accelerators. Embedding dimensions and patch sizes shape execution patterns, making some models inherently better matched to specific hardware. Memory configuration also scales with model size: smaller models perform well with 64KB local and 2MB global buffers, while larger models like CLIP-L/14 and CLIP-H/14 require at least 128KB local buffers and larger global memory.

0.5

0.5

0.5

carbon-100ms

0.4

0.4

0.4

latency

(%)

(%)

energy

0.3

0.3

0.3

20 TOPS

4 TOPS

0.2

0.2

0.2

carbon-100ms

carbon-100ms

0.1

latency

0.1

latency

0.1

0.0

6.3

0.4

0.5

0.6

energy

0.7

0.8

0.0

6.3

0.4

0.5

0.6

energy

0.7

0.8

0.0

6.3

0.4

0.5

0.6

1 TOPS

0.7

0.8

Carbon (kgCO2e)

Carbon (kgCO2e)

Carbon (kgCO2e)

4.4 Accuracy Evaluation

We compare CarbonCLIP models to TinyCLIP and the CLIP-ViT-B/16 baseline (pretrained on DataComp-1B), using each baseline's most carbon-efficient configuration. Table 3 shows the CarbonCLIP family, selected from the Pareto frontiers in Appendix E. For each CarbonCLIP model, We perform post-pruning training, and evaluate their performance on 41 zero-shot tasks from the CLIPBenchmark [LA22]. The table summarizes model and hardware configurations for CarbonCLIPXL to CarbonCLIP-XS (largest to smallest models), along with their carbon footprint, latency, and average accuracy. Full per-dataset results are in Appendix H. We also extend the CarbonCLIP family to the CLIP-B/32 architecture for comparison with TinyCLIP-B/32 baselines (Appendix G).

CarbonCLIP models outperform baselines across a range of carbon budgets. Notably, CarbonCLIP-XL achieves baseline-level accuracy with an 10% reduction in carbon footprint. CarbonCLIP-XS achieves an 8% increase in accuracy with a 3% reduction in carbon footprint compared to TinyCLIP-8M/16. CarbonCLIP-L, CarbonCLIP-M, and CarbonCLIP-S all achieve significant reductions in carbon footprint compared to TinyCLIP-39M/16, with CarbonCLIP-L achieving a 4% increase in accuracy and a 4.5% reduction in carbon footprint, CarbonCLIP-M achieving an 11% reduction in carbon footprint with a 3% decrease in accuracy, and CarbonCLIP-S achieving a 17% reduction in carbon footprint without any regression in accuracy.

In terms of hardware configuration, CarbonCLIP models select accelerators with cores of PE x dimension 256 to align with the underlying operator dimensions of the CLIP ViT-B/16 architecture, with sequence length of 197 for the vision encoder. Smaller models select a total of 1024 PE units per core, whereas larger models select twice as many PEs to keep the latency of the task low. Due to the reduced size of the CarbonCLIP models, a 64KB local memory and 2MB global memory are sufficient to keep the core utilized.

4.5 Evaluating for Different Compute Constraints

We evaluate CarbonCLIP models under three peak compute constraints inspired by real-world edge devices: 20 TOPS [HAI25, Nvi25], 4 TOPS [Goo20], and 1 TOPS [WLS + 24, Int25]. Figure 5 shows the resulting Pareto frontiers.

Takeaway 6: As hardware shrinks due to tighter compute constraints, optimizing for energy (operational carbon) increasingly minimizes both total carbon and latency.

Smaller hardware designs reduce embodied carbon, making operational energy the dominant factor in the total carbon footprint. As a result, energy-optimized designs often achieve outcomes similar to those optimized for total carbon under tight compute budgets. Appendix I explores the embodied vs. operational carbon trade-offs in more detail. Note that different grid intensities and expected hardware lifetime can also affect the ratio of embodied and operational carbon, affecting the optimization results, and the quality of each metric immensely, we show this in Appendix K. However, smaller accelerators have limited compute and memory, which increases inference delay-and thus total energy use-despite lower power consumption. This leads to higher total carbon emissions in some cases, underscoring the importance of hardware-model co-design to balance carbon efficiency and performance. We further extended the study to evaluate the impact of latency constraints on model and hardware configurations, as well as their carbon footprints in Appendix J.

5 Impact Discussion and Limitations

This work demonstrates a practical path toward sustainable AI by jointly optimizing model and hardware design for carbon reduction. Our framework is particularly effective for low-latency edge inference (e.g., chatbots, AR/VR) and highlights the potential for cross-domain collaboration across ML, hardware, and sustainability.

CATransformers currently targets Transformer models and domain-specific edge accelerators, with limited GPU support and no evaluation on CNNs. Extending to other model families (e.g., CNNs, Mamba) requires profiling the latency and energy of their unique operators and adapting pruning strategies beyond attention heads. Expanding to GPUs and data center scenarios is feasible, but the larger design space would demand significantly more search. Nonetheless, our GPU-based results (Appendices L and M) suggest scalability to training workloads and datacenter-level accelerators, where support for parallelization and distributed training could further improve carbon efficiency.

Beyond inference and training emissions, broader sustainability impacts, such as electronic waste, water usage, and rare mineral consumption, remain underexplored. While our optimizations can implicitly reduce chip area and resource use, standardized metrics and fine-grained data are currently lacking, limiting multi-objective optimization over broader environmental impacts. Moreover, custom accelerators raise concerns about hardware heterogeneity and e-waste, which may be mitigated by reusing existing hardware or co-optimizing with commercial accelerators.

Proxy-based accuracy estimation, such as the methods employed in CATransformers, enables scalable exploration but may deviate from real outcomes due to overfitting or underestimating pruning effects. Developing more robust accuracy predictors is an important direction for improving reliability.

6 Conclusion

We introduced CATransformers, a framework for co-optimizing model architectures and domainspecific accelerators to minimize carbon footprint. By jointly considering operational and embodied carbon, CATransformers supports environmentally conscious design, particularly in edge computing. We demonstrated its effectiveness across various Transformer-based models-including multi-modal models-showcasing substantial potential in carbon reductions without sacrificing performance. This work fills a key gap in sustainable AI deployment and provides a foundation for future research in carbon-aware machine learning.

References

[ALK + 23] Bilge Acun, Benjamin Lee, Fiodar Kazhamiaka, Aditya Sundarrajan, Kiwan Maeng, Manoj Chakkaravarthy, David Brooks, and Carole-Jean Wu. Carbon dependencies in datacenter design and management. SIGENERGY Energy Inform. Rev. , 3(3):21-26, October 2023.

[APK + 24] Muhammad Adnan, Amar Phanishayee, Janardhan Kulkarni, Prashant J. Nair, and Divya Mahajan. Workload-aware hardware accelerator mining for distributed deep learning training, 2024.

[App21] Apple Inc. Apple environmental progress report 2021. https://www.apple.com/ environment/pdf/Apple_Environmental_Progress_Report_2021.pdf , 2021.

[BBL23] Lucía Bouza, Aurélie Bugeau, and Loïc Lannelongue. How to estimate carbon footprint when training deep learning models? a guide and review. Environmental Research Communications , 5(11):115014, nov 2023.

[bJRE + 23] colby banbury, Vijay Janapa Reddi, Alexander Elium, Shawn Hymel, David Tischler, Daniel Situnayake, Carl Ward, Louis Moreau, Jenny Plunkett, Matthew Kelcey, Mathijs Baaijens, Alessandro Grande, Dmitry Maslov, Arthur Beavis, Jan Jongboom, and Jessica Quaye. Edge impulse: An mlops platform for tiny machine learning. In D. Song, M. Carbin, and T. Chen, editors, Proceedings of Machine Learning and Systems , volume 5, pages 254-268. Curan, 2023.

[BKJ + 20] Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G Wilson, and Eytan Bakshy. Botorch: A framework for efficient monte-carlo bayesian optimization. Advances in neural information processing systems , 33:2152421538, 2020.

[CHY + 21] Kanghyun Choi, Deokki Hong, Hojae Yoon, Joonsang Yu, Youngsok Kim, and Jinho Lee. Dance: Differentiable accelerator/network co-exploration. In 2021 58th ACM/IEEE Design Automation Conference (DAC) , pages 337-342, 2021.

[CKES16] Yu-Hsin Chen, Tushar Krishna, Joel S Emer, and Vivienne Sze. Eyeriss: An energyefficient reconfigurable accelerator for deep convolutional neural networks. IEEE journal of solid-state circuits , 52(1):127-138, 2016.

[CSL + 24] Benoit Courty, Victor Schmidt, Sasha Luccioni, Goyal-Kamal, MarionCoutarel, Boris Feld, Jérémy Lecourt, LiamConnell, Amine Saboni, Inimaz, supatomic, Mathilde Léval, Luis Blanche, Alexis Cruveiller, ouminasara, Franklin Zhao, Aditya Joshi, Alexis Bogroff, Hugues de Lavoreille, Niko Laskaris, Edoardo Abati, Douglas Blank, Ziyao Wang, Armin Catovic, Marc Alencon, Michał St˛ echły, Christian Bauer, Lucas Otávio N. de Araújo, JPW, and MinervaBooks. mlco2/codecarbon: v2.4.1, May 2024.

[DBB21] Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Parallel bayesian optimization of multiple noisy objectives with expected hypervolume improvement. In Proceedings of the 35th International Conference on Neural Information Processing Systems , NIPS '21, Red Hook, NY, USA, 2021. Curran Associates Inc.

[DBK + 21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021.

[DCLT19] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pretraining of Deep Bidirectional Transformers for Language Understanding, 2019.

[DGC24] Dayou Du, Gu Gong, and Xiaowen Chu. Model quantization and hardware acceleration for vision transformers: A comprehensive survey, 2024.

[ECA + 23] Mariam Elgamal, Doug Carmean, Elnaz Ansari, Okay Zed, Ramesh Peri, Srilatha Manne, Udit Gupta, Gu-Yeon Wei, David Brooks, Gage Hills, et al. Carbon-efficient design optimization for computing systems. In Proceedings of the 2nd Workshop on Sustainable Computer Systems , pages 1-7, 2023.

[ECA + 25] Mariam Elgamal, Doug Carmean, Elnaz Ansari, Okay Zed, Ramesh Peri, Srilatha Manne, Udit Gupta, Gu-Yeon Wei, David Brooks, Gage Hills, and Carole-Jean Wu. CORDOBA: Carbon-Efficient Optimization Framework for Computing Systems . In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA) , pages 1289-1303, Los Alamitos, CA, USA, March 2025. IEEE Computer Society.

[FKW + 24] Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Osi, Prateek Sharma, Fan Chen, and Lei Jiang. Llmcarbon: Modeling the end-to-end carbon footprint of large language models, 2024.

[Fu24] Yao Fu. Challenges in deploying long-context transformers: A theoretical peak performance analysis, 2024.

[GDJ + 24] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and et al. The llama 3 herd of models, 2024.

[GEH + 22] Udit Gupta, Mariam Elgamal, Gage Hills, Gu-Yeon Wei, Hsien-Hsin S. Lee, David Brooks, and Carole-Jean Wu. Act: designing sustainable computer systems with an architectural carbon modeling tool. In Proceedings of the 49th Annual International Symposium on Computer Architecture , ISCA '22, page 784-799, New York, NY, USA, 2022. Association for Computing Machinery.

[GHL + 25] Shreyank N. Gowda, Xinyue Hao, Gen Li, Shashank Narayana Gowda, Xiaobo Jin, and Laura Sevilla-Lara. Watt for what: Rethinking deep learning's energy-performance relationship. In Computer Vision - ECCV 2024 Workshops: Milan, Italy, September 29-October 4, 2024, Proceedings, Part XXII , page 388-405, Berlin, Heidelberg, 2025. Springer-Verlag.

[GIF + 23] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets, 2023.

[GKL + 22] Udit Gupta, Young Geun Kim, Sylvia Lee, Jordan Tse, Hsien-Hsin S. Lee, Gu-Yeon Wei, David Brooks, and Carole-Jean Wu. Chasing carbon: The elusive environmental footprint of computing. IEEE Micro , 42(4):37-47, July 2022.

[GKX + 24] Soroush Ghodrati, Sean Kinzer, Hanyang Xu, Rohan Mahapatra, Byung Hoon Ahn, Dong Kai Wang, Lavanya Karthikeyan, Amir Yazdanbakhsh, Jongse Park, Nam Sung Kim, and Hadi Esmaeilzadeh. Tandem processor: Grappling with emerging operators in neural networks. In ASPLOS , 2024.

[Goo20] Google. Helping you bring local ai to applications from prototype to production, 2020.

[HAI25] HAILO. Hailo-8 ai accelerator, 2025.

[Har25] Harmony Healthcare IT. Are you addicted to your phone? american phone usage & screen time statistics. https://www.harmonyhit.com/phone-screen-time-sta tistics/ , 2025.

[HYW + 23] Shiqi He, Qifan Yan, Feijie Wu, Lanjun Wang, Mathias Lécuyer, and Ivan Beschastnikh. Gluefl: Reconciling client sampling and model masking for bandwidth efficient federated learning. In D. Song, M. Carbin, and T. Chen, editors, Proceedings of Machine Learning and Systems , volume 5, pages 695-707. Curan, 2023.

[HZRS16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2016.

[IME25] IMEC. imec.netzero. " https://netzero.imec-int.com/" , 2025.

[Int25] Intel. Intel movidius myriad x vision processing unit, 2025.

[IWW + 21] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. If you use this software, please cite it as below.

[JKL + 23] Norman P. Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Cliff Young, Xiang Zhou, Zongwei Zhou, and David Patterson. Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings, 2023.

[JYP + 17] Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre-luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati, William Gulland, Robert Hagmann, C. Richard Ho, Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexander Kaplan, Harshit Khaitan, Daniel Killebrew, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, and Doe Hyun Yoon. Indatacenter performance analysis of a tensor processing unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture , ISCA '17, page 1-12, New York, NY, USA, 2017. Association for Computing Machinery.

[JYS + 20] Weiwen Jiang, Lei Yang, Edwin Hsing-Mean Sha, Qingfeng Zhuge, Shouzhen Gu, Sakyasingha Dasgupta, Yiyu Shi, and Jingtong Hu. Hardware/software co-exploration of neural architectures. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , 39(12):4805-4815, 2020.

[KMH + 20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.

[Kri09] Alex Krizhevsky. The cifar-10 dataset, 2009.

[LA22] LAION-AI. Clip benchmark, 2022.

[LBL + 24] Haokun Lin, Haoli Bai, Zhili Liu, Lu Hou, Muyi Sun, Linqi Song, Ying Wei, and Zhenan Sun. Mope-clip: Structured pruning for efficient vision-language models with module-wise pruning error metric. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 27360-27370, 2024.

[LGH + 24] Sasha Luccioni, Boris Gamazaychikov, Sara Hooker, Régis Pierrard, Emma Strubell, Yacine Jernite, and Carole-Jean Wu. Light bulbs have energy ratings - so why can't ai chatbots? Nature , 632:736-738, 08 2024.

[LJGT24] Baolin Li, Yankai Jiang, Vijay Gadepally, and Devesh Tiwari. Sprout: Green generative AI with carbon-efficient LLM inference. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 21799-21813, Miami, Florida, USA, November 2024. Association for Computational Linguistics.

[LJS24] Sasha Luccioni, Yacine Jernite, and Emma Strubell. Power hungry processing: Watts driving the cost of ai deployment? In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency , FAccT '24, page 85-99, New York, NY, USA, 2024. Association for Computing Machinery.

[LLSD19] Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700 , 2019.

[LMB + 15] Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015.

[LYH21] Yujun Lin, Mengtian Yang, and Song Han. Naas: Neural accelerator architecture search. In 2021 58th ACM/IEEE Design Automation Conference (DAC) , pages 1051-1056, 2021.

[Map25] Electricity Maps. The leading electricity grid api, 2025.

[MP24] Inc. Meta Platforms. Adaptive experimentation platform, 2024.

[MPA + 16] Divya Mahajan, Jongse Park, Emmanuel Amaro, Hardik Sharma, Amir Yazdanbakhsh, Joon Kyung Kim, and Hadi Esmaeilzadeh. Tabla: A unified template-based framework for accelerating statistical machine learning. In 2016 IEEE International Symposium on High Performance Computer Architecture (HPCA) , pages 14-26. IEEE, 2016.

[NSY24] NSYS Group. Average device lifespan: How long does a cell phone last? https: //nsysgroup.com/blog/average-device-lifespan-how-long-does-a-cel l-phone-last , 2024.

[Nvi25] Nvidia. Jetson orin nano, 2025.

[ONFL23] MohammadHossein Olyaiy, Christopher Ng, Alexandra Fedorova, and Mieszko Lis. Sunstone: A Scalable and Versatile Scheduler for Mapping Tensor Algebra on Spatial Accelerators. In 2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) , 2023.

[PMK + 22] Amar Phanishayee, Divya Mahajan, Janardhan Kulkarni, Miguel Castro, and Muhammad Adnan. Workload-aware hardware architecture recommendations, 2022. US Patent App. 17/965,681.

[PMT25] Amar Phanishayee, Divya Mahajan, and Jakub Michal Tarnawski. Integrated hardware architecture and distribution strategy optimization for deep learning models, 2025. US Patent App. 18/452,162.

[PRS + 19] Angshuman Parashar, Priyanka Raina, Yakun Sophia Shao, Yu-Hsin Chen, Victor A. Ying, Anurag Mukkara, Rangharajan Venkatesan, Brucek Khailany, Stephen W. Keckler, and Joel Emer. Timeloop: A Systematic Approach to DNN Accelerator Evaluation. In 2019 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) , pages 304-315, 2019.

[PSM + 17] Jongse Park, Hardik Sharma, Divya Mahajan, Joon Kyung Kim, Preston Olds, and Hadi Esmaeilzadeh. Scale-out acceleration for machine learning. In Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture , pages 367-381, 2017.

[QPFM + 23] Xinchi Qiu, Titouan Parcollet, Javier Fernandez-Marques, Pedro P. B. Gusmao, Yan Gao, Daniel J. Beutel, Taner Topal, Akhil Mathur, and Nicholas D. Lane. A first look into the carbon footprint of federated learning. J. Mach. Learn. Res. , 24(1), January 2023.

[RKH + 21] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.

[SBV + 22] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models, 2022.

[SDDN20] Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. Poor man's BERT: smaller and faster transformer models. CoRR , abs/2004.03844, 2020.

[SGM20] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for modern deep learning research. Proceedings of the AAAI Conference on Artificial Intelligence , 34(09):13693-13696, Apr. 2020.

[SJZ + 20] Ananda Samajdar, Jan Moritz Joseph, Yuhao Zhu, Paul Whatmough, Matthew Mattina, and Tushar Krishna. A systematic methodology for characterizing scalability of dnn accelerators using scale-sim. In 2020 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) , pages 58-68. IEEE, 2020.

[SSL23] Chirag Sakhuja, Zhan Shi, and Calvin Lin. Leveraging domain information for the efficient automated design of deep learning accelerators. In 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA) , pages 287-301, 2023.

[STJ + 23] Dachuan Shi, Chaofan Tao, Ying Jin, Zhendong Yang, Chun Yuan, and Jiaqi Wang. UPop: Unified and progressive pruning for compressing vision-language transformers. In Proceedings of the 40th International Conference on Machine Learning , volume 202, pages 31292-31311. PMLR, 2023.

[SVB + 21] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. ArXiv , abs/2111.02114, 2021.

[SWL + 22] H. Ekin Sumbul, Tony F. Wu, Yuecheng Li, Syed Shakib Sarwar, William Koven, Eli Murphy-Trotzky, Xingxing Cai, Elnaz Ansari, Daniel H. Morris, Huichu Liu, Doyun Kim, Edith Beigne, Reality Labs, and Meta. System-level design and integration of a prototype ar/vr hardware featuring a custom low-power dnn accelerator chip in 7nm technology for codec avatars. In 2022 IEEE Custom Integrated Circuits Conference (CICC) , pages 01-08, 2022.

[SZW + 18] Ananda Samajdar, Yuhao Zhu, Paul Whatmough, Matthew Mattina, and Tushar Krishna. Scale-sim: Systolic cnn accelerator simulator. arXiv preprint arXiv:1811.02883 , 2018.

[vMVJ22] Jakub Žádník, Markku Mäkitalo, Jarno Vanne, and Pekka Jääskeläinen. Image and video coding techniques for ultra-low latency. ACM Comput. Surv. , 54(11s), September 2022.

[VPF + 24] Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, and Oncel Tuzel. Mobileclip: Fast image-text models through multi-modal reinforced training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15963-15974, 2024.

[WDS + 19] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's transformers: State-of-the-art natural language processing. CoRR , abs/1910.03771, 2019.

[WES19] Yannan Nellie Wu, Joel S. Emer, and Vivienne Sze. Accelergy: An ArchitectureLevel Energy Estimation Methodology for Accelerator Designs. In 2019 IEEE/ACM International Conference on Computer-Aided Design (ICCAD) , pages 1-8, 2019.

[WLS + 24] Tony F. Wu, Huichu Liu, H. Ekin Sumbul, Lita Yang, Dipti Baheti, Jeremy Coriell, William Koven, Anu Krishnan, Mohit Mittal, Matheus Trevisan Moreira, Max Waugaman, Laurent Ye, and Edith Beigné. 11.2 a 3d integrated prototype system-on-chip for augmented reality applications using face-to-face wafer bonded 7nm logic at< 2 µ m pitch with up to 40% energy reduction at iso-area footprint. In 2024 IEEE International Solid-State Circuits Conference (ISSCC) , volume 67, pages 210-212, 2024.

[WNM23] Irene Wang, Prashant J. Nair, and Divya Mahajan. FLuID: Mitigating stragglers in federated learning using invariant dropout. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

[WPZ + 23] Kan Wu, Houwen Peng, Zhenghong Zhou, Bin Xiao, Mengchen Liu, Lu Yuan, Hong Xuan, Michael Valenzuela, Xi (Stephen) Chen, Xinggang Wang, Hongyang Chao, and Han Hu. Tinyclip: Clip distillation via affinity mimicking and weight inheritance. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 21970-21980, October 2023.

[WRG + 22] Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, Michael Gschwind, Anurag Gupta, Myle Ott, Anastasia Melnikov, Salvatore Candido, David Brooks, Geeta Chauhan, Benjamin Lee, Hsien-Hsin Lee, Bugra Akyildiz, Maximilian Balandat, Joe Spisak, Ravi Jain, Mike Rabbat, and Kim Hazelwood. Sustainable ai: Environmental implications, challenges and opportunities. In D. Marculescu, Y. Chi, and C. Wu, editors, Proceedings of Machine Learning and Systems , volume 4, pages 795-813, 2022.

[WSM + 18] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. CoRR , abs/1804.07461, 2018.

[WTPM24] Irene Wang, Jakub Tarnawski, Amar Phanishayee, and Divya Mahajan. Integrated hardware architecture and device placement search. In International Conference on Machine Learning , 2024.

[WWL + 20] Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. Hat: Hardware-aware transformers for efficient natural language processing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, 2020.

[XXT + 24] Hu Xu, Saining Xie, Xiaoqing Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying CLIP data. In The Twelfth International Conference on Learning Representations , 2024.

[ZDM + 22] Yanqi Zhou, Xuanyi Dong, Tianjian Meng, Mingxing Tan, Berkin Akin, Daiyi Peng, Amir Yazdanbakhsh, Da Huang, Ravi Narayanaswami, and James Laudon. Towards the co-design of neural networks and accelerators. In D. Marculescu, Y. Chi, and C. Wu, editors, Proceedings of Machine Learning and Systems , volume 4, pages 141-152, 2022.

[ZG23] Yiyang Zhao and Tian Guo. Carbon-efficient neural architecture search. In Proceedings of the 2nd Workshop on Sustainable Computer Systems , HotCarbon '23, page 1-7. ACM, July 2023.

[ZHS + 22] Dan Zhang, Safeen Huda, Ebrahim Songhori, Kartik Prabhu, Quoc Le, Anna Goldie, and Azalia Mirhoseini. A full-stack search technique for domain optimized deep learning accelerators. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems , page 27-42, New York, NY, USA, 2022. Association for Computing Machinery.

[ZLJG24] Yiyang Zhao, Yunzhuo Liu, Bo Jiang, and Tian Guo. CE-NAS: An end-to-end carbonefficient neural architecture search framework. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

Appendix

A Ablation Study: Pruning and Finetuning CLIP models

In this ablation study, we studied the effect of pruning each dimension and the effect of fine-tuning on various datasets to find a good proxy for approximating the accuracy after training the pruned models on larger datasets fir CLIP-based models. All models were evaluated against MS COCO. The results are shown in Figure 6 and our key observations are as follows:

We make some key observations in terms of the importance of each dimension to the overall accuracy of the model. First, accuracy drops significantly after pruning to 50% of any dimension, even after fine-tuning. Therefore, we confine the search space to a minimum of half of each dimension. Second, the vision model has a more significant impact on accuracy compared to the text model. Finally, pruning the embedding dimension has the most significant impact on accuracy among all pruned parameters, followed by number of layers, then FFN dimensions and number of attention heads.

Fine-tuning on MS COCO: Even with just a single epoch, fine-tuning on MS COCO significantly improves the accuracy of pruned models, making it a good and fast proxy for evaluating their overall potential. Fine-tuning on Datacomp-Tiny: Training on a small subset of a pre-training dataset (Datacomp-Tiny, a 400k subset of Datacomp-Medium unfiltered) also improves accuracy, albeit with lower overall accuracy compared to MS COCO. Fine-tuning on Datacomp-Medium: Using a general pre-training dataset (Datacomp-Medium unfiltered) reduces the variance in accuracy between models of different sizes, showing that smaller models can achieve comparable accuracy when trained with enough data. This incentivizes our post-pruning training with a large and high quality MetaCLIP 2.5B Dataset.

In general, larger models will attain higher accuracy compared to models with fewer parameters when trained for the same number of steps. However, models with fewer parameters may still achieve comparable accuracy as larger models given more training steps [KMH + 20]. Therefore, we fine-tune each model with the same computation FLOPS, allowing smaller models to train for more FLOPS and recover their accuracy from more extensive pruning.

Text Layer

Text FFN

Text Embedding Dim

Text Attn Heads

none

mscoco  datacomp-tinydatacomp-M

none

mscoco

datacomp-tinydatacomp-M

none

?

mscoco

datacomp-tiny  datacomp-M

none

mscoco

datacomp-tiny  datacomp-M

Average Accuracy

0.8

0.6

0.8

0.6

80

0.6

0.8

0.6

0.2

0.4

Avg

0.2

0.4

0.4

0.2

0.4

0.2

0.0

0.5

0.6

0.7

8'0

6°0

1.0

00

0.5

0.6

0.7

0.8

6°0

1.0

0.0

0.5

0.6

0.7

0.8

60

1.0

0.0

0.5

0.6

0.7

80

60

1.0

% of Layers

% of FFN dimension

% Embedding Dimension

% Attn Heads

Vision Layer

Vision Embedding Dim

Vision FFN

Vision Attn Heads

none

?

datacomp-tiny datacomp-M

none

mscoco

datacomp-tiny  datacomp-M

none

mscoco

datacomp-tiny datacomp-M

none

mscoco

datacomp-tiny

datacomp-M

Average Accuracy

0.8

0.6

0.8

0.6

0.8

0.6

0.8

0.6

0.2

0.4

Average

0.4

0.2

Average

0.4

0.2

Average

0.4

0.2

0'0

0.5

0.6

0.7

0.8

0.9

1.0

0.0

0.5

0.6

0.7

0.8

0.9

1.0

0.0

0.5

0.6

0.7

0.8

0.9

1.0

0.0

0.5

0.6

0.7

0.8

0.9

1.0

% of Layers

% Embedding Dimension

% FFN Dimension

% Attn Heads

B Estimation Tool-chain Integration

In this section we present in detail the integration of the tools used in the hardware estimator as illustrated in Figure 7. We use Accelergy [WES19] to estimate the area and access energy of each component, and Sunstone [ONFL23] to estimate the per-operator latency and energy. For carbon estimations, we use ACT [GEH + 22] to estimate the embodied carbon of the hardware architecture based on the area of the accelerator. Given the energy estimates from Sunstone, Electricity Maps [Map25] is used to retrieve the carbon intensity of the electricity of a given grid location and calculate the operational carbon of executing a single inference. We then scale the operational carbon to the total lifetime of the hardware architecture.

HW

Architecture

Operator

graph

Torch.FX

Accelergy

Sunstone

ACT

Area

Access

Energy

Energy

Embodied

Carbon

Latency

Electricity

Maps

Operational

Carbon

Per architecture

estimation

Per operator

estimation

Embodied carbon

estimation

Carbon Intensity

estimation

Input

Tools

Intermediate Output

Output

Cacti

Proxy(MSCOCO)

Final(Average of41datasets)

60

Accuracy (%)

40

20

0

C Correlation Between Proxy and Final Model Accuracy

To efficiently approximate accuracy while preserving ranking consistency, we fine-tune each pruned model to a common dataset used to evaluate each model type. As shown in Figure 8, fine-tuning maintains a high Spearman's rank correlation coefficient (0.98) with the final post-pruning training accuracy, ensuring reliable accuracy ranking. We report the mean top-1 recall accuracy on MSCOCO as the accuracy proxy for each pruned CLIP model. We incorporate this into the optimization loop to dramatically reduce the cost of evaluating candidate model architectures.

D ISO Accuracy

In this sections we present full details on different design points found by each optimization metric for a given accuracy point in Tables 4, 5, 6, and 7, for CLIP-ViT-B16, BertBase, Llama3-8b, ViT-B16, respectively. The key observations are discussed in Section 4.2. For each iso-accuracy comparison, accuracy values are reported with a tolerance of +/- 1% across optimization strategies.

E Pareto Frontier of each Optimization

In this section, we present an example of the Pareto frontier for each optimization metric using the CLIP ViT-B/16 model. Figure 9 shows the Pareto frontiers for latency-only, carbon-only, energy-only, and latency+carbon optimization objectives. Each data point in the figure represents a specific model and hardware architecture configuration, with accuracy shown on the y-axis, carbon footprint on the

Total

Carbon

Total

Carbon +

Latency

Energy

Latency

0.40

0.05

0.40

0.05

0.40

0.05

0.40

0.05

0.35

0.35

0.35

0.35

0.30

0.04

0.30

0.04

0.30

0.04

0.30

0.04

0.25

0.20

0.03

Latency

0.25

0.20

0.03

atency

Accurae

0.25

0.20

0.03

atency

Accura

0.25

0.20

0.03

Latency

0.15

0.02

0.15

0.02

C

0.15

0.02

0.15

0.02

0.10

0.01

0.10

0.01

0.10

0.01

0.10

0.01

0.05

0.05

0.05

0.05

0.00

8.3

0.4

0.5

0.6

0.7

0.8

0.00

0.00

6.3

0.4

0.5

0.6

0.7

0.8

0.00

0.00

0.3

0.4

0.5

0.6

0.7

0.8

0.00

0.00

0.3

0.4

0.5

0.6

0.7

0.8

0.00

Carbon (kgCO2e)

Carbon (kgCO2e)

Carbon (kgCO2e)

Carbon (kgCO2e)

x-axis, and latency encoded via a color map. To ensure consistency, each experiment is repeated three times, and accuracy is estimated using the MS COCO dataset. When latency is not an explicit optimization objective, a maximum latency constraint of 50 ms is enforced [vMVJ22] to ensure realistic deployment scenarios.

Consistent with the discussion in Section 4.2, examining the Pareto frontiers reveals that optimizing solely for total carbon yields configurations with the lowest carbon footprint across all accuracy levels. However, this comes at the cost of higher latency compared to the other optimization objectives.

F Consistency of Experimental Results

Each optimization is repeated three times for fair evaluation. To assess consistency, we compute the Hypervolume (HV) indicator across runs with the same configuration. HV measures the portion of the objective space dominated by the Pareto front, relative to a reference point. Across all modelhardware optimizations, the standard deviation of HV remains below 0.03, with an average coefficient of variation (std/mean) under 3.5%, indicating stable and reproducible results. Table 8 summarizes these statistics for each model architecture and optimization objective.

G CLIP ViT B/32 Accuracy Evaluations

In this section, we present results that demonstrate the applicability of CATransformers to the CLIP ViT-B/32 architecture and compare its performance against the TinyCLIP baselines. (Table 9) Our findings show that CATransformers's optimization can be effectively generalized to other model architectures, yielding up to 5% and 8% reductions in carbon footprint while achieving higher accuracy and comparable latency compared to the TinyCLIP baselines, respectively.

H CLIP Benchmark Full Result

In this section we provide a detailed breakdown on the accuracy of each dataset for CarbonCLIP and the evaluated baselines in Table 10.

I Carbon Footprint Breakdown

We provide a breakdown of the carbon footprint for each variant of the CarbonCLIP family model. As shown in Figure 10, as the model increases in size, the proportion of operational carbon in the overall carbon footprint of the model increases from 20% to over 40%. The CarbonCLIP-XL model has 3 × the number of parameters and almost 3 × the latency of CarbonCLIP-XS, but the selected hardware architecture only has double the number of compute PEs. Therefore, the operational carbon increases proportionally more than the increase in embodied carbon. This highlights need of co-optimizing the model and hardware architecture to maintain an intricate balance between operational and embodied carbon, keeping the overall carbon footprint of the system low. As such, the expected lifetime and source of power are also important factors that need to be taken into account during the optimization process.

Operational(total lifetime)

■Embodied

100%

75%

50%

25%

0%

XS

S

M

L

XL

AverageAccuracy(%)

0.40

carbon-100ms

0.35

carbon-50ms

carbon-10ms

0.30

0.25

0.20

0.15

0.10

0.05

0.00

0.3

0.4

0.5

0.6

0.7

0.8

Carbon (kgCO2e)

J Case Study: Varying Latency Constraints

We evaluate the impact of latency constraints on model and hardware configurations, as well as their carbon footprints. We categorize use cases into three categories based on latency requirements: critical real-time ( < 10ms), interactive ( < 50ms), and non-critical ( < 100ms) [vMVJ22]. Figure 11 shows the Pareto frontiers for each category.

Lower latency constraints typically result in less carbon-efficient designs. For example, a 10ms latency constraint achieves only an 17% carbon reduction compared to latency-optimized models, although with comparable latency values. However, increasing the constraint to 100ms does not consistently improve efficiency, as many optimal designs already meet the 50ms threshold.

K Case Study: Varying Operational Regions

The region of the operation and manufacturing of the model affects the search results, and the quality of the metric immensely. Figure 12 shows that varying the operational carbon region, in high, mid, low carbon-intensity regions, yields dramatically different results in terms of the model searched and resulting configurations. Additionally, while our case studies (Section 4.5) show that energy optimization is effective when operational carbon dominates (e.g., smaller architectures, high-carbonintensity regions), in cleaner energy regions, total carbon optimization still yields lower emissions. Optimizing against total carbon instead of energy can reduce emissions by 8% in low-carbon regions (Canada) vs. 2% in high-carbon regions (Taiwan). The pareto frontier of each region is shown in Figure 13a and Figure 13b.

0.5

Taiwan

CA,USA

0.4

BC, Canada

0.3

62

curae

0.2

0.1

0.0

6.3

0.4

0.5

0.6

0.7

0.8

Carbon (kqCO2e)

(a) British Columbia (Low carbon intensity)

(b) Taiwan (High carbon intensity)

0.5

Carbon

Energy

0.4

(%)

0.3

ccuracy

0.2

Ac

0.1

5780

0.30

0.35

0.40

0.45

0.50

Carbon(kqCo2e)

0.5

Carbon

Energy

0.4

0.3

ccuracy

0.2

AC

0.1

0.8.3

0.4

0.5

0.6

0.7

0.8

Carbon (kqC02e)

L Case Study: Model Architecture Search with GPUs

Once a hardware architecture is selected, CATransformers supports model architecture search on fixed hardware to refine configurations and re-evaluate carbon costs before manufacturing.

Table 11 shows an example of model architecture search on a fixed V100-like hardware architecture, optimizing total carbon and latency.

M Energy and Latency Validation with GPUs

To maintain high confidence in our estimation toolchains, we validated our energy and latency estimates using existing GPU hardware (V100, A100, and H100). Although GPUs differ from

Explored Model Architectures

Energy (J)

0

1

2

3

Estimation Energy - Scaled

Measured Energy

(a) V100 - Energy

Explored Model Architectures

Latency (s)

0.00

0.01

0.02

0.03

0.04

Estimated Latency - Scaled

Measured Latency

(b) V100 - Latency

Explored Model Architectures

Energy (J)

1.0

1.5

2.0

2.5

Estimated Energy - Scaled

Measured Energy

(c) A100 - Energy

Explored Model Architectures

Latency (s)

0.0100

0.0125

0.0150

0.0175

0.0200

Estimate Latency - Scaled

Measured Latency

(d) A100 - Latency

Explored Model Architectures

ENergy (J)

0

1

2

3

Estimated Energy - Scaled

Measured Energy

(e) H100 - Energy

Explored Model Architectures

Latency (s)

0.0050

0.0075

0.0100

0.0125

0.0150

Estimated Latency - Scaled

Measure Latency

(f) H100 - Latency

domain-specific accelerators, we modeled a GPU-like architecture with comparable tensor and vector units, conducted a model search on the fixed architecture, and profiled actual hardware performance for each searched data point.

The results demonstrate a strong correlation between our estimates and measured performance, with Spearman's rank-order correlation ranging from 0.5 to 1.0. When energy and latency values are scaled to a common measurement range using constants, the estimates yield average errors of 8% for energy and 9% for latency across GPU architectures. This validation confirms that our estimation tools provide accurate and reliable results, even with simulated hardware. Table 12 summarizes the results, while Figure 14 presents the estimated and measured performance for each evaluation GPU architecture.

N Carbon Footprint of the Framework

We quantify the carbon footprint of running CATransformers using CodeCarbon [CSL + 24]. On average, the optimization process takes 5 hours for BERTbase and up to 20 hours for CLIP models. For CLIP, the most resource-intensive case, 100 optimization trials emit approximately 57 kgCO2e, while final model training emits 454 kgCO2e per model. This means the optimization process costs roughly 1/13th the carbon budget of training the final model. Despite the one-time cost of optimization, CATransformer achieves overall efficiency gains through reduced training steps post-pruning, along with inference gains that scale with the number of devices.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All claims made in the abstract and introduction are substantiated within the paper. An overview of the framework and its design details is provided in Section 3, while key insights, analysis, and empirical results related to carbon optimization are presented in the evaluation sections, particularly in Section 4.2. Training results for the CarbonCLIP model and comparisons to prior work are discussed in Sections 4.3 and 4.4. Additional implementation details and ablation studies are included in the Appendix.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We included a Discussion and Limitations (See Section § 5).

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Paper does not include theoretical proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper provides a detailed overview of all framework components in Section 3, including search parameters and integration methodology. Additional information on the estimation toolchain is provided in Appendix B to ensure full reproducibility. The experimental setup, covering evaluation baselines, dataset and model sources, as well as constants and assumptions used, is outlined in Section 4.1. Furthermore, the source code is included as supplementary material for reference.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The source code is included as part of the supplementary material and will be open-sourced upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/pu blic/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experimental setup details, including all evaluated models, datasets used, and hyperparameters, are provided in Section 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Spearman's rank correlation coefficients are reported to establish the statistical significance of fine-tuning proxies (Section 3.2, Appendix C) and GPU validation

results (Appendix M). For the Bayesian Optimization results in Section 4.2, each experiment is repeated three times to ensure consistency. We demonstrate the reproducibility of the optimization outcomes by evaluating the Hypervolume (HV) indicator across runs of the same configuration, and report the standard deviation and coefficient of variation for each model and optimization objective in Appendix F. Estimation error rates for latency and energy are discussed in Section 4.1, with additional details provided in Appendix M. Accuracy deviations relevant to ISO-accuracy comparisons are clearly reported in Figure 4 in Section 4.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Section 4.1 provides details for the compute platform, memory, and time required for experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Authors have reviewed the NeurIPS Code of Ethic, and all submitted material preserves anonymity.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss both the limitation and impact of the work in Section 5.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not pose such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper cites all relevant prior work, including frameworks, codebases, datasets, and pretrained models used in both the design and evaluation. Detailed references are provided in Section 3, Section 4.1, and the Appendix.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Source code is included as part of the supplementary material (anonymized), with detailed instructions on how to run and reproduce the experiments.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: Core method development in this research does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces CATransformers, a carbon-aware co-optimization framework that jointly searches Transformer architectures and custom accelerator designs to minimize total lifecycle carbon emissions (operational + embodied). The method couples (i) a multi-objective Bayesian optimizer (qNEHVI on Ax/BoTorch) balancing accuracy, latency, energy, and carbon, (ii) a lightweight prune-and-fine-tune evaluator that supplies high-rank-correlated proxy accuracies, and (iii) a hardware estimator that integrates Accelergy, Sunstone, ACT, and Electricity Maps to quantify area, energy, latency, and carbon for candidate accelerators early in design space exploration.\nExperiments span BERT-Base, Llama-3 (8 B), ViT-B/16, and multi-modal CLIP variants. With a 20 TOPS edge budget, carbon-optimized solutions cut total emissions by up to 30 % over latency-optimized and 8 % over energy-optimized baselines, at comparable accuracy. A focused case study yields the CarbonCLIP family, which achieves up to 17 % lower carbon than TinyCLIP while matching or improving accuracy/latency. The framework is to be open-sourced.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Introduces a novel, holistic framework that co-optimizes both ML models and hardware accelerators to directly minimize total lifecycle carbon emissions (operational + embodied). Novel combination of Bayesian NAS with joint embodied + operational carbon objective; extends prior work (e.g., CE-NAS) from model-only to full stack. Multi-modal CLIP optimisation under carbon constraints is new.\n2. First study to treat carbon as first-class search objective across model and hardware; could influence sustainable ML practice on edge devices. CarbonCLIP models are likely reusable by practitioners focusing on low-footprint deployments.\n2. The authors cross-check latency/energy estimates against real GPUs (≤ 9 % error), show a Spearman 0.98 correlation between proxy- and full-accuracy, and report low hyper-volume variance (< 3.5 %) across search runs, evidence that both the estimator and optimizer are reliable.\n3. Empirical results across diverse architectures (BERT, Llama-3, ViT, CLIP) demonstrate up to 30 % emission cuts over latency-optimized baselines, with the CarbonCLIP family delivering ~17 % lower carbon while maintaining accuracy.\n4. Clear visualizations (Pareto fronts, ISO-accuracy plots) effectively communicate trade-offs and design insights.\n\nWeaknesses:\n1. Restrictive carbon assumptions i.e. fixed 1 inf/s duty cycle, 3-year lifetime, California grid intensity, and Taiwan 22 nm fabrication limit generalizability; alternative scenarios are only briefly treated in the appendix.\n2. estimator details (e.g., specific Accelergy/Sunstone knobs, ACT parameters) are not fully documented, which may hinder independent reproduction\n3. targets edge-inference accelerators; training-time carbon and data-center/cloud inference remain unaddressed.\n4. several tables are hard to parse, some abbreviations go undefined, and embodied-carbon scaling equations (fundamental to the paper) are omitted from the main text.\n5. There are several works [1-5] that try to answer sustainability related issues in AI as well as estimate CO2 footprint. I think listing them would benefit the paper and give a more holistic understanding of how researchers have been trying to tackle this problem before.\n\n[1] Bouza, L., Bugeau, A. and Lannelongue, L., 2023. How to estimate carbon footprint when training deep learning models? A guide and review. Environmental Research Communications, 5(11), p.115014.\n[2] Luccioni, S., Jernite, Y. and Strubell, E., 2024, June. Power hungry processing: Watts driving the cost of ai deployment?. In Proceedings of the 2024 ACM conference on fairness, accountability, and transparency (pp. 85-99).\n[3] Luccioni, S., Gamazaychikov, B., Hooker, S., Pierrard, R., Strubell, E., Jernite, Y. and Wu, C.J., 2024. Light bulbs have energy ratings—so why can’t AI chatbots?. Nature, 632(8026), pp.736-738.\n[4] Gowda, S.N., Hao, X., Li, G., Gowda, S.N., Jin, X. and Sevilla-Lara, L., 2025. Watt for what: Rethinking deep learning’s energy-performance relationship. In European Conference on Computer Vision (pp. 388-405). Springer, Cham.\n[5] Strubell, E., Ganesh, A. and McCallum, A., 2020, April. Energy and policy considerations for modern deep learning research. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 09, pp. 13693-13696).\n\nQUESTIONS:\n1. Results assume 1 inference/s for 3 years. How would conclusions change for higher-throughput mobile scenarios (e.g., 10 inf/s) or shorter device lifetimes? A sensitivity plot in the main paper would strengthen applicability.\n2. Appendix K touches on regional grids, but design choices might invert (energy vs. latency) under a zero-carbon grid. Can the optimizer ingest a distribution over grid intensities and return robust designs? Clarify in 3.3/4.5.\n3. You plan to open-source, but key configs (e.g., Cacti SRAM models, yield factors) are unspecified. Please list exact tool versions and parameter files to enable artifact evaluation.\n4. The evaluator fine-tunes each candidate for one epoch on MS-COCO/GLUE etc. What is the wall-clock overhead per candidate, and is it amortized across search iterations? A brief cost breakdown would help readers gauge scalability.\n5. Carbon is central, but other sustainability axes (e-waste, water, mineral use) may correlate with chip area. Could CATransformers be extended to multi-impact objectives? A short discussion in the conclusion would increase societal relevance.\n\nScores could rise with clearer reporting of (1)–(3) and an expanded robustness study for different duty cycles / grids.\n\nLIMITATIONS:\nAuthors acknowledge lack of GPU/CNN support, focus on edge inference, and fixed carbon-intensity/lifetime assumptions. Suggested to foreground these in  Sec 5 and add an explicit paragraph on potential negative societal impacts beyond carbon (privacy, resource inequity).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduced CATransformers which is a carbon-aware search framework for sustainability-driven co-optimization of ML models and hardware architectures. They tackled minimizing the total carbon emission including both embodied and operational by jointly optimizing for model's and hardware accelerator's design, and their approach includes three main parts: 1) a multi-objective bayesian optimizer that maximizes accuracy while minimizing latency, energy, and total carbon (embodied + operational) by exploring different model and hardware parameter, 2) ML model evaluator which estimates the accuracy of candidate model architectures during the search, 3) Hardware estimator that provides unified, end-to-end analysis of inference latency and total carbon footprint for each model–hardware configuration. The final output is the sustainable model and hardware configuration.\n\nThe authors evaluated their proposed approach on various models like BERT, Llama, etc. They show their approach can reduce carbon emissions by up to 30% while maintaining accuracy and latency. They also showed that their optimized CLIP model outperformed models like TinyCLIP and baseline CLIP in accuracy and total carbon emissions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) Comprehensive experiments that show their approach can perform in various scenarios\n2) The paper is well-written and organized and informative by having key takeaways for each experiment\n3) The authors also consider embodied carbon as a parameter for total emissions which is a critical parameter\n4) Novel approach to jointly optimize model and hardware arch. from the base when the objective is carbon reduction which tells us the optimization should start from early stages.\n\nWeakness:\n1) The current approach is tailored for edge devices and inference where the carbon optimization in large-scale clusters, e.g., HPC clusters can have greater impact\n2) There is no clear discussion section on what the challenges could be of extending the architectures from transformer-based models to another architecture\n3) While the Pareto frontiers are informative, it could benefit from more intuitive interpretations, e.g., how practitioners should choose between latency and carbon optimization in real deployments remains a bit underspecified.\n\nQUESTIONS:\n1) Can you please clarify what the challenges of using other model archs like CNN could be?\n2) Since there is a trade-off between latency and carbon consumption, do you have an approach to help the user select which configuration tailored to their need?\n3) Is it possible to plug-in CATransformer into PyTorch? what are the challenges?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CATransformers, a novel co-design framework that jointly optimizes Transformer-based model architectures and hardware accelerators with carbon efficiency as a primary objective. It addresses the growing carbon footprint of AI systems by incorporating both operational and embodied carbon into the early-stage design space exploration. The framework integrates a multi-objective Bayesian optimizer, a structured pruning-based model evaluator with proxy fine-tuning, and a hardware estimator that quantifies carbon, latency, and energy. Experiments across language, vision, and multi-modal models demonstrate CATransformers’ ability to reduce total carbon emissions without sacrificing accuracy or latency, highlighting its effectiveness through the design of CarbonCLIP, a family of low-carbon CLIP variants.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. This paper is the first to systematically incorporate carbon emissions as a primary optimization objective in the co-design of Transformer architectures and hardware accelerators. \n\n2. The proposed CATransformers is designed to be reproducible and adaptable across a wide range of Transformer models.\n\n3. The paper systematically analyzes trade-offs across multiple objectives, model scales, and hardware configurations.\n\n\nWeakness:\n\n1. Some components in the method section are described rather compactly, which may hinder reproducibility without the code release.\n\n2. The current framework exclusively targets inference optimization. It does not account for training-phase carbon cost, especially from large-scale pretraining of foundation models. And this may dominate the total carbon footprint. This limits the general applicability of the framework.\n\n3. While the method aims to reduce model carbon impact, the carbon cost of the search process itself is not measured or discussed.\n\nQUESTIONS:\n1. The current framework focuses solely on inference-time optimization — does this neglect the potentially much larger carbon cost of training, especially for large-scale pretraining?\n\n2. While the method reduces model carbon footprint, could the carbon cost of the search process itself be substantial and offset the benefits?\n\nLIMITATIONS:\nWhile the paper does offer some discussion of its limitations, I would suggest that it could be further strengthened by considering the following aspects.\n\n1. The authors should discuss how to balance the potential sustainability challenges introduced by custom accelerators, such as increased electronic waste and maintenance costs due to hardware heterogeneity.\n\n2. The authors could briefly discuss the risks of proxy-based estimations deviating from real-world outcomes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a model-architecture and hardware co-design framework for jointly optimizing priority metric like carbon footprint (both embodied and operation) followed by latency, energy and accuracy. A new open source hardware evaluation tool chain for design space exploration is also showcased. The framework is evaluated across multiple directions with join hardware and model architecture improvement, ablations with hardware or model architecture fixed. The work provides a basis for important decision-making in choosing co-optimized hardware and model architecture for sustainability focused end-users.  Co-optimized model–hardware configurations reduce total carbon by 30% over latency optimized and 8% over energy-optimized baselines.\n\nSTRENGTHS AND WEAKNESSES:\n+ Good paper on framing the groundwork for identifyng design tradeoffs for large models\n+ Good use of light-weight fune tuning as a proxy for accuracy during Bayes Opt.\n+ The open-source toolchain is really useful for holistic design space exploration for latency, energy and total carbon emission\n+ It shows faster/high volume evaluation rate (100M/100iters)\n+ Pruned models require less training budget than full-pretraining.(60% reduction)\n+ The ablations and the corresponding takeaways also help gain great insights in to model architecture and hardware design choices,\n+ The works highlights the complex trade-offs that were not previously considered in earlier works\n\n- None that I can identify\n\nQUESTIONS:\n* Based on the discussion in section 3.1: Does it mean that the carbon footprint of the pretrained model can't be optimized since we start from it and run the pruning. Or, is the suggestion that if we pretrain on the pruned model from scratch, we will recover the same accuracy? (this is in realation to the discussion in last line of 3.5)\n\nLIMITATIONS:\nNone that I could identify since I am not an expert in co-design area.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-30"}