From Words to Wires: Generating Functioning Electronic Devices from Natural Language Descriptions
Authors: Peter Jansen
Venue: EMNLP 2023 Findings (2023)
Keywords: applications
code generation
electronics
language models
TL;DR: Demonstrates that language models like GPT-4 can design electronic circuits from high-level textual descriptions, achieving high success rates in generating 25 benchmark devices, while functioning as a design assistant for more complex devices.

Abstract: In this work, we show that contemporary language models have a previously unknown skill -- the capacity for electronic circuit design from high-level textual descriptions, akin to code generation. We introduce two benchmarks: PINS100, assessing model knowledge of electrical components, and MICRO25, evaluating a model's capability to design common microcontroller circuits and code in the Arduino ecosystem that involve input, output, sensors, motors, protocols, and logic -- with models such as GPT-4 and Claude-V1 achieving between 60% to 96% Pass@1 on generating full devices. We include six case studies of using language models as a design assistant for moderately complex devices, such as a radiation-powered random number generator, an emoji keyboard, a visible spectrometer, and several assistive devices, while offering a qualitative analysis performance, outlining evaluation challenges, and suggesting areas of development to improve complex circuit design and practical utility.  With this work, we aim to spur research at the juncture of natural language processing and electronic design.

FROM WORDS TO WIRES: Generating Functioning Electronic Devices from Natural Language Descriptions
Peter Jansen
University of Arizona pajansen@arizen izona...ii.. edu
Abstract
"Design random number generator that uses ambient radiation level to set the random seed, and displays random numbers nixie tube."
this work, we show that contemporary lan- guage models have previously unknown skill the capacity for electromic circuit design from high-level textual descriptions, akin to code generation. We introduce two benchmarks: PINS100, assessing model knowledge of elec- trical components, and MICRO25, evaluating a model's capability to design common miero- controller circuits and code the ARDUINO ecosystem that involve input, output, sensors, motors, protocols, and logic- bisic-wwiie with models such GPT-4 and Cl achieving hetween 60% to 96% PASS @ on generating full de- vices. We include six case studies of using language models
Large Language Model
design assistant for mod- erately complex devices, such as radiation- powered random number generator, an emoji keyboard, visible spectrometer, and several assistive devices, while offering qualitative analysis performance, outlining evaluation chal- lenges, and suggesting areas of development to improve complex circuit design and practical utility. With this work, we aim spur research the juncture of natural language processing and electronic design." 12
example of using a language model convert a high-level textual description of an electronic device into the designs for that device. Those designs then reviewed by a domain expert, and any errors corrected, before the device is physically manufactured using rapid-prototyping techniques.
Figure 1:
Flam-Shepherd and Aspuru-Guzik, 2023), and gen- erate increasingly complex sections of code that power a variety of user applications (Li et al., 2023; Wang et al., 2023).
1 Introduction
The realm of science fiction often presents us with captivating visions of technology's ss future. A case in point is the replicator from the TV series Star Trek, a machine capable of creating various physi- cal objects from food and medicine function- ing devices based solely on a user's high-level description of those objects. Contemporary lan- guage models hint at the precursors to some of this capacity for design, including the ability to design novel 2D and 3D object models (Ramesh et al., 2022; Nichol et al., 2022), predict molecu- lar structures for drug discovery (Liu et al., 2021;
In this work, we show that language models have previously unknown skill the capacity to gen- erate working electronic devices from high-level textual descriptions effectively bridging the gap between the words of a device description to the wires of device design. The design process for electronic devices, such as the random number generator in Figure 1, typically follows stage- like process illustrated in Figure 2. These steps include: ideation, electronic design (including gen- erating parts lists, electronic schematics, and code for embedded processors called microcontrollerss. followed ultimately by physical implementation
|PYTHON library and data: cognitiveailabe wtt.vveeaaaa/wodddwwrrrrr https://github.cco
*Companion video: https://youtt: http:://yaatubee///rrrddarr be/PZIT

---

Words2Wires (this work)
Physical Implementation
"Nixie-Tube Random Number Generator"
901
Idea
Bill of Materials
Component Pinouts
Schematics
Microcontroller Code
PCB Design
Construction
Figure 2: An overview the electronics device design process, from concept to design implementation.
via manufacturing. Our focus in this work is automating the task of transforming a high-level concept into practical electronics schematic, com- plete with companion microcontroller code-a task that currently requires significant human expertise and effort. We evaluate our approach by construct- ing these devices either in simulators or, for six open-ended case studies, physical devices im- plemented using rapid prototyping techniques such breadboards, circuit board manufacture, laser cutting, and 3D printing.
2 Related Work
Arduino microcontroller ecosystem: This work focuses on circuits that are controlled by microcon- trollers, which are processors intended for embed- ded applications. In particular, we focus on micro- controllers supported by the ARDUINO ecosystem (Banzi and Shiloh, 2022), popular cross-platform set of C++ tools and libraries intended to pro- mote learning and lower the barrier entry for creating physical devices. Most of the devices de- scribed this work target the ARDUINO UNO, a platform with more than 10 million units sold as of 2023, which uses an ATMEGA328P microcon- troller with 2K of RAM, 32K of program space, 20 input/output (I/O) pins, and a speed of 16MHz. While these specifications (e.g. RAM and speed) modest for desktop computers, they are typi- cal of microcontrollers and embedded applications where programs are typically run bare-metal (i.c. without an operating system).
The contributions in this work are:
1. We empirically demonstrate the novel capac- ity for language models to design electronic devices from high-level textual descriptions, and introduce two benchmarks to measure this capacity. The first, PINS measures knowl- edge of 100 common electronic components. The second, MICRO25, measures the ability to design 25 common electronic devices from start to finish, including the generation of bills- of-materials, pinouts, schematics, and code.
Though aspects of physical design (such as circuit board layout) have been automated for decades (see Huang et al. (2021) for review), to the best of our knowledge, this is the first work use language models for automating early-stage circuit design, converting high-level textual device descriptions into initial electronic schematics and microcontroller code. Designing code-driven electronic circuits is similar to code generation tasks (c.g. Chen et al., 2021; Austin et al., 2021; Hendrycks et al., 2021; Lai et al., 2022: Nijkamp et al., 2023, inter alia), with the additional requirement that model must jointly generate corresponding electrical circuit schematic that is compatible with the code, that together allow the microcontroller to accomplish a given task.
Code Generation:
2. Our experimental evaluation shows that both GPT-4 and CLAUDE-Y V1 have moderate-to- strong performance on these benchmarks, with GPT-4 achieving 96% PASS@ 1 at gener- ating correct schematics and functioning code on MICRO25, while CLAUDE-V V1 scores 60% schematics and 76% code.
3.
We present six real-world case studies of us- ing language models to construct novel de- vices, including an emoji keyboard, a visible light spectrometer, and two assistive devices. Alongside, provide qualitative assess- ment of the strengths of current language mod- els for electronic device design, while also outlining the challenges in enhancing perfor- mance, automating evaluation, and increasing their practical utility.
Conditioned Code Generation: Generating code from intermediate planning representations (such
Phttps:/ ttttp:://wwwr //www//wwwww/// arduino..cm cc/

---

"Create device that turns on an LED when a button is pressed"
Textual Device Description
// controlled by button
AAddunr Uno'; "name" Yuno"" "valus?"" "nutes AArduin Uno microcontraller') F'prrt "part"Button" name":5W1T "value"?", Pnotes""Momentan
//hhi code interfaces with circuit that has single push button an LED.
// The LED turs on when the button is pressed, and turns off when the button not pressed
button").
adufine PIN BUTTONS
Button Digital LOO LD Digital VO
('part/LED" "name""D1" "valuc" "notes"" Standard voltage range (2-3.3V):). I'part Resistor", "namees "value"" "notes"
#define LED
setup
when you press power
limiting resistor for LED at SV")
setup01
pinModa(PINN BUTTON, PULLUPE:
Iiiitiiaize button pin input mode (using
internal pull up resisto)
Bill of Materials
pinMode/PINN LED, OUTPUT)
If Initialize LED pin to output mode
the loop function runs over and over forever
Uno's ("M": "3,3"" "GND! "D6/RX;; 'DS//T;; "D27 103", "D4, "DS "D6""""'" "D?; "DB; "09"; "D117 "D122, "D13". "AO", 'AR", 'A2", "As", "AA/SDA;, (AS/SCL"). "SW1'E""" *21), "DT's "Dn's"anode"" ""aaooo""cattooee""" 11111
digitalReadd@"1 BUTTON); Read
dggtta//tte//iN.ccc LED, HIGHT
3 lese
I/ the button not pressed, turn off the LED
digitalWrte/PIN.LEE. LOW;
Component Pinouts
Microcontroller Code
D1
RI
Arduino Uno
(SW',"" "pnnt"1" C' d'name": "uno'; "omoo//ppnn""" "pin'D3" (SWn"""WWi" W "pan"22", 2 "unom""""nn""" "pin""GND"1
e Connect Button pin Uno Connect Button pin to GND
U'name'R11 RR", "p'n's'1's E'name's "uno"; DD5'1.
lcurrent limiting resistor) aetivate/deactivvee limiting resistor)
UO3
SWI
I'namerD11, pinnr'nndee' Phname 3RRI", "pinietrng ["nameeDD "al""cathode's I'hame't "uno", "ae""NND")
pin
GHD
Schematics IVisual Representation)
Schematics as Annotated JSON (Netlists, as output by the LUMI
Language Model Generated
00 JPC
Simulated Construction
Physical Construction
Figure An example of representing a device specification as (in formatted JSON) such that it can be generated by language model, for trivial device that illuminates LED in response to a button being pressed. device specification includes the bill of materials, component pinouts, schematic (represented as netlist), and microcontroller code. The specification can then be used to create that device, either in simulation through physical construction. The device shown here was generated by GPT-4, and edited slightly for space.
UML diagrams) increase task performance (Liu et al., 2022b), and here we adapt this to the electronics context by generating device specifi- cations (such as schematics) immediately before code generation, to condition code generation on a specific device. Similarly, generating structured representations (in the form of code) can better elicit the knowledge in language model (Madaan al., 2022), where here adapt this generating device specifications highly-structured JSON representations. These device specifications (bills of materials, pinouts, schematics, and code) are thousands of tokens long, and their generation is enabled by the increase in model context lengths to
8K tokens, allowing models to generate hundreds of lines of code (OpenAI, 2023; Li al., 2023) compared to earlier models with smaller genera- tion capacity (Chen et al., 2021; Li et al., 2022: Fried et al., 2023).
Hardware Description Languages: A contem- poraneous body of work describes code genera- tion tasks for hardware description languages (such as VHDL VERILOG) that run specialized processors called Preer-Programmmalle Gate Ar- rays (FPGAAs) typically prototype specialized CPU designs, or accelerate digital signal process- ing. Thakur et al. (2023) introduce a benchmark of 17 simple bitwise tasks, with the most advanced

---

tasks including bitwise addition or counting, and show the best performing model (CODEGEN-6B) to achieve 60% PASS@ 10 when solving these tasks. Similarly, Blocklove et al. (2023) use GPT-4 as a Verilog coding assistant, demonstrating that it can perform well at generating 8 simple Verilog designs (such bitwise adder, or 3-state finite state machine), while also providing qualitative evalu- ation of using GPT-4 assistant design an 8-bit accumulator-basee microprocessor. con- trast, where these projects generate code that runs on FPGAs, this work (WORDS2W WIRES) generates both electrical schematics and code for electrical devices that are built from and interface with real electrical components, such as sensors, motors, and displays, while also providing a larger-scale set of benchmarks that perform significantly more complex and real-world tasks.
GPT3.5
GPT-4
Claude
Pinout Scoring Method
Striet Scoring (Fxact)
55%
74
56%
Permissive Scoring
74%
86 %
72%
Table I: Model accuracy in generating accurate pinout information for 100 common electronic components. Strict scoring requires all generated pins on given device to be accurate in order to be counted as correct. Permissive scoring requires pins critical to the function of a given component to be correct, but still counts generations with non-critical missing pins (such as when the device is mounted to breakout board) be correct.
While LEDs and other com- mon components such as resistors and capacitors commonly have two pins, other components have varying numbers of pins, each with different func- tions. For example, a common kind of digital motor called a hobby servo typically has three pins: an an- ode, a cathode, and a digital signal pin that accepts a pulse from a microcontroller that signifies what angle the motor should turn to. Common sensors generally have between two and ten pins, depend- ing the communication protocols (e.g. digital vs analog) they employ. Similarly, integrated cir- cuits, and microcontrollers in specific, can have few as 8 pins (such the ArTINY85 microcon- troller in Figure 2). but frequently have dozens and occasionally hundreds of unique pins. Connect- ing component pins incorrectly a schematic will cause a device to malfunction, so this knowledge is critical to constructing working circuits.
Component Pinouts:
Single-shot Collaborative: In this work, investigate device generation in two contexts. First, we assess single-turn generation of error-free de- vices using the MICRO25 benchmark, a set of clec- tronic design tasks evaluated using code metrics similar to PASS@) (Kulal et al., 2019) that require generating a single correct solution, with strict bi- nary measures of task success. In the second con- text, we explore collaborative coding-assie assistant setting, where prompts can be iteratively refined, and any errors in the schematics or code corrected by the end user. This approach is akin to tools such as GITHUB COPILOT (Chen et al., 2021), which assist in generating short parts of programs. How- ever, in our case, the model is utilized to generate an initial version of the entire project, after which the user corrects any errors before physically con- structing the device.
We assembled a benchmark of clec- tronic component pinouts, PINS 100, containing 100 common parts frequently used in circuits found on high-traffic electronic tutorial websites such as the ARDUINO PROJECT HUB and AU- TODESK TINKERCAD CIRCUITS. Components range from 2 to 40 pins, and span a large assort- ment of part categories including passives (c.g. re- sistors/capacitors) input (e.g. switches), output (e.g. LEDs, motors, relays), sensors, integrated cir- cuits, power regulators, logic (e.g. 7400-SERIES AND and OR gates), and microcontrollers (e.g AR- DUINO, RASPBERRY Pt).
Benchmark:
3 Experiment 1: Component Knowledge
To design electronic circuits, the designer needs a knowledge of the individual electrical components that can be used to build a circuit. One of the most fundamental aspects of this knowledge is the component pinouts, or the specific function of each electrical terminal (or pin) on component. For example, light-emitting diode (LEDD typically anode where positive voltage
has two pins, is applied, and one cathode where the negative terminal or ground is applied. Here we measure large language models' knowledge of component pinouts by asking them to generate pinouts for a large number of common electrical components.
We evaluate on instruction-tuned mod- els including OpenAl's CHATGPT (GPT-3.5- tURBO) and GPT-4 (OpenAI, 2023), and An- thropic CLAUDE-VIL V Model prompts are identi-
Models:
"https://www http://www..attrrrri..oom

---

cal across models, and include static 1-shot ex- emplar (a 14-pin 7400-SERIES logic integrated circuit) that provides an example of the pinout task, as well as the requested JSON output format. Addi- tional hyperparameters, evaluation details, and the full prompt are provided in the APPENDIX.
to be programmed onto a microcontrollerr light- weight processor that controls embedded circuits. An example of complete model-generated design for a trivial device that turns on an LED in response to a button being pressed is shown in Figure 3.
Benchmark: To assess a model's ability to cre- ate microcontroller-drive electronic devices, we developed a benchmark, MICRO25, that includes 25 tasks intended for the common ARDUINO mi- crocontroller ecosystem.. These tasks, shown in Table 2, span 5 core categories including: input, in- terface protocols, output, sensors, and logic. Each task is either tailored to test specific fundamental competency required to build basic microcontroller- driven electronic devices common in introductory microcontroller curricula, or the integration of sev- eral competencies into larger design flows.
Evaluation: We evaluate using two binary mea- sures of accuracy analogous to the PASS @ code- generation metric (Kulal al., 2019). The first scoring method, strict, requires given model to output all of a component's pins correctly to be considered correct, otherwise will be considered incorrect. The second method, permissive, requires only the function-crities pins of a component to be present to be considered correct, while failing to include non-critical pins still counts success."
Results: Model performance in the pinout gener- ation task is shown in Table 1. Performance re- flects average binary PASS I performance a given model on generating accurate pinouts - for example, a score of 50% reflects that 50% of the components had completely correct pinouts. Here, GPT-4 achieves the highest strict scoring perfor- mance, generating accurate pinouts for 74% of components, while both GPT3.5 and CLAUDE-V achieve similar levels of performance, generating correct pinouts for 55% and 56% of components, respectively. Permissive scoring increases perfor- mance, with the best-scoring GPT-4 model capable of generating pinouts that include the most critical pins for 86% of electrical components in the bench- mark. Taken together, these results suggest that large language models have a moderate-to-stron knowledge of electrical component pinouts, core requirement for designing functioning electronic circuits.
Representations: Models were given format prompts to export all generated elements (bill of materials, pinouts, schematics, and code) in an an- notated JSON format, shown in Figure 3. The anno- tated format allows the model to add comments for each generated element (e.g. specifying the uses of each component, or the purpose of each connection in the schematic), analogous to chain-of-thoo tf--thuughtt reasoning (Wei et al., 2022) applied to circuit gener- ation, as well code generation from requirements specifications (Liu et al., 2022b), where here the requirements are the schematics and other device specifications generated immediately preceding the code. Additional details of this representation for- mat are provided in APPENDIX B.4 and C.
Models: We evaluate on instruction- tuned models with large (8k token) context windows, including OpenAl's GPT-4 (OpenAI, 2023) and Anthropic CLAUDE-VA V1. Prompts are identical across both models, and include a static minimal 1-shot exam- ple of generating each of the 4 elements of a device specification (bill of materials, pinouts, schematic, code) in the desired JSON output format. In sponse to specific types of errors identified during pilot studies, the prompt also includes three incom- plete snippets that provide portions of two positive and one negative generation example. After ini- tial generation, the models are given a reflection
4 Experiment 2: Circuit Generation
How well can contemporary language models lever- age their component knowledge design simple but functioning electronic devices? In this ex- periment, we investigate end-to-end generation of working devices, which includes generating four core elements: (1) a bill of materials (BOM), or list of components in the device, (2) the pinouts for each component, (3) complete electrical circuit diagram called schematic that details how the components are be connected, and (4) the code
MMCRO225 and PINS 100 require ovvellaping component knowledge. Solving MICRO25 requires some (but not all) of the components in PINS Similarly, each MICRO25 task has many possible solutions, and with hun- dreds of thousands of different electrical components currently available, it is possihle to generate solutions whose schematies use one or more components not listed in PINS 100.
'Adiitional details of permissive scoring (including an examplo) can he found in APPENDIX B.

---

Schematic (GPT4)
Schematic (Claude)
Code (GPT-)
Code (Claude)
Category
Task Description
Input
Digital Button
Turn on an I/O pin when a single button is pressed.
Digital Multiple
Turn on an I/O pin when exactly of buttons are pressed. Read potentiometer, activate I/O pin when voltage exceeds 2.5V
Analog
x
Protocols
Serial/UART
Read the serial port. When "hello" is read, respond with "world". Connect using SPI. When "hello" is read, respond with "world". Read value from specific I2C address and register every second. Open file, randomly append one 4 strings every seconds.
Scrial/SPI
12C SD Card
Output
Motor - DC
Rotate a DC motor clockwise then counterclockwise every 5 seconds. Oscillate stepper motor 45 degrees every seconds.
Motor Stepper
Motor - Servo
Continuously move servo back and forth from 45 to 135 degrees. Blink an LED every 500 milliseconds.
LED Blink
LED Sequence
Blink 4 LEDs in sequence. once every 500 milliseconds.
X
LED 7 Segment
Count from 0 to on a 7-segment display, changing every 500 msec.
LED Neopixel
an RGB LED. Turn on relay for seconds, then off for 5 seconds.
Slowly move through rainbow of colours
Relay
X
LCD
Print "Hello World" a 16x2 LCD (HF44778 compatible controller) Cycle between high, medium, then low tones on Pieza Buzzer. Produce sawtooth wave, ramping up from OV to 5V every 50 msec.
Sound Buzzer
Analog Outpur
X
Sensors
Resistive - CDS Manual Protocol
Read the light intensity from resistive sensor (CDS cell). Read the distance from HC-SR04 ultrasonic distance sensor, Read x/y/z components of magnetic field using an I2C magnetometer.
12C - Magnetic
Logic
Simon
Create the popular memory game Simon, using 4 colors.
4
4
X
Conway
Create the popular Conway's Game of Life, 8x8 LED matrix. Create clock a 16x2 I2C LCD, with buttons for setting the time. Read the temperature, display temperature an RGB LED.
X
Clock
Air Temperature
/
Overall Performance
96%
96
611%
76%
Table Model performance (PASS @ 1) on the MICRO25 benchmark generation tasks, broken down by schematic and code. Task descriptions are summarized for space, where full task descriptions can be found in APPENDIX E.
prompt containing 12 common errors (such cor- rectly supplying power each component, explic- itly enumerating each connection in the schematic, and having code that functions as intended), and allowed to iteratively reflect and improve output until providing a specific stop token signifying that the model has detected no further errors. The ini- tial prompt requires 1884 tokens, and the reflec- tion prompt requires 431 tokens. Additional model details, including evaluation details and the full prompt are provided in the APPENDIX.
Because the schematic subsumes the bill of ma- terials and component pinout information, we do not evaluate the BOM or pinouts independently, only the entire schematic - but do still include generating the BOM and pinouts in the prompt to facilitate chain-of -of-thoughtttt reasoning. Due to the challenges of automatic evaluation in this domain, evaluation was conducted manually by domain expert through inspection, simulation, and physical circuit construction.
Results: The results of the device generation task on the MICRO25 benchmark are shown in Table 2. Across all tasks, model-generated code ranged
Generated devices are broken down into electrical (schematic) and code components, each of which is separately evaluated using a binary PASS metric (i.e. functional or non-funt non-functional).
Evaluation:
from minimum of 13 lines to a maximum of 145 lines (average 38 lines per program). GPT-4 per-

---

forms extremely well on the MICRO25 benchmark, correctly generating schematics and code for 96% of benchmark tasks. CLAUDE-V V1 exhibits more modest performance, achieving 60% for schematic generation, and 76% for microcontroller code gen- eration. Taken together, this shows that contempo- rary language models have moderate-to-excelle overall capacity for generating common electrical circuits end-to-end, from bills of materials, pinouts, and schematics, to paired microcontroller code that accomplishes the desired functionality.
quired by domain expert to reach functionality are described in the APPENDIX.
Generation challenges can be organized into two categories: prompt- specific challenges, and hardware-specific chal- lenges. With respect prompts, device generation is highly sensitive to the specific prompt, and small (and seemingly helpful) changes the prompt to address an error can cause new errors to occur in other aspects of the device that were previously gen- erated correctly. Similarly, having many composite requirements in the project (such as adding the re- quirement for each key in the emoji keyboard to generate its own musical tune) generally decreases the capacity to design comparatively simple de- vices in Experiment 2. this result is tempered by these benchmark tasks being representative of fairly common skills and capacities that are frequently taught in watprccoonntolle-orieettte curricula found in books, internet tutorials, and blog posts and such, the MICRO25 tasks likely exist in some form in the voluminous (but closed) training data of these models. Here, we examine how well the best- perf-rerfon model, GPT-4, can create compar- atively more complex devices in a more realistic and qualitative setting, where it is used as de- sign assistant like GITHUB COPILOT (Chen et al.. 2021) to create initial plans that are then vetted and corrected by domain expert before being physi- cally constructed. To further increase task difficulty, each of the device specifications in this experiment were explicitly crafted to be highly unusual - ci- ther using uncommon components, or combining common components in unusual ways such that the likelihood of similar devices appearing in the closed model training data is low. The six devices are shown in Figure 4.
Qualitative Challenges:
5 Experiment 3: Open Device Generation
While we've observed that language models have
performance, and suggests that iteratively gener- ating devices from the core requirements (such generating a functioning emoji keyboard) through to more fine-grained details (like adding in musical tunes) may reduce the inference load at each gener- ation step, improving generation performance.
With respect to hardware, number of pragmatic issues occur. Electronic parts regularly reach end- lf-l cycles and are no longer manufactured or easily available, yet the model frequently generates these, likely due to the abundance of examples that use these components in internet tutorials. Sim- ilarly, the model frequently uses deprecated ver- sions of libraries, or combines the features from different versions of libraries. Finally, the model generally performs poorly at generating low-level device drivers for specific hardware (such sen- sors), and favors using existing device driver li- braries. When an existing library isn't available. the model will either hallucinate one, or generate a reasonable first-pass at a device library that re- quires extensive modification to low-level details (like clock timings) to function.
Methods: Initial specifications (in the form of natural language textual description of a device) were created for all devices, and iteratively refined several times to provide clarifications in response to undesired or errorful model-generated output. Af- ter several attempts at refining device descriptions, any remaining errors were manually corrected by a domain expert, then the devices were physically manufactured. All devices were constructed by a human, and physical aspects of the design (e.g. printed circuit boards, 3D printed or laser cut com- ponents) were designed by a human. Full device description strings, model prompts, and detailed qualitative descriptions of manual corrections re-
Challenges and Discussion
We identify the following challenges and oppor- tunities in developing this capacity for automated device design further:
Prompt Sensitivity: the prompt can cause large changes in the output, affecting overall performance. While this is evi- dent from small changes in task description strings producing novel errors in Experiment this phe- nomenon is also visible for simpler cases. For ex- ample, the pinouts for some components such as the MLX906 14 temperature sensor - are incorrect when tested independently in Experiment 1, but
Currently, small changes in

---

Random number generator that uses ambient radiation sensor to continuously update the random seed. Random numbers are generated every few seconds, and displayed o vintage nixie tube using high-voltage driver.
Emoji USB keyboard that has keys for 9 common emojis. Pressing an emoji types its ASCII string, just as if entered on normal keyboard. A short musical tune with similiar affect to the emoji (e.g. love song for the heart emoji) also played.
Visible light spectrometer that measures spectra using Hamamatsu miicr--pectromeete and displays the spectrum on 128x128 pixel OLED screen. Here, the device is shown measuring the characteristic spectrum white LED.
Non-contact temperature sensor that displays the temperature on strip of LEDs. Higher temperatures show as red colors and illuminate more LEDs, while lower temperatures show blue colors and illuminate fewer LEDs.
TAKE PILL
Time: 6436404
Pill alarm assistive device that has alarms for three pills. When it's time take pill, the alarm continuously waives a physical flag saying "Take Pill X" back and forth using servo motor to get the users attention, until the button is pressed.
Ultrasonic glasses assistive device that
ultrasonic
distance sensor measure the distance of the nearest object in front of a person with a visual impairment. The distance is converted into an audible tone of varying frequency.
Figure 4: Six devices designed using WORDS2WIRES in the open generation condition, then physically constructed.
correct when generated as part of a full circuit in Experiment 2. This suggests that task performance is not currently robust, and may benefit from creat- ing and fine-tuning on a task-specific dataset.
guage models are rarely perfect and frequently have errors. Currently these have be discovered and corrected by human. Contemporary work in code generation aims to use reflection (e.g. Shinn et al., 2023) to iteratively run generated code an ex- ternal interpreter (like PYTHON), report any errors
Manual Evaluation: Like COPILOT, we have ob- served that the electronic designs generated by lan-

---

the language model, then continue this process until the generated code runs error-free. The lack of electronic simulators with large libraries of sim- ulated devices presents a significant barrier to this form of automatic evaluation in the near-term. Ul- timately this may be addressed by constraining circuit generation to only parts available within a given simulator, or pushing a focused effort to developing more capable simulators with larger repertoire of components.
7 Conclusion
This study empirically characterizes the previously unknown potential of contemporary language mod- els to move from words to wires - that is, to gen- erate working electronic device designs from high- level text descriptions. Our analysis demonstrates these models have moderate-to-higgh proficiency in generating component-level knowledge on the PINS 100 benchmark, while GPT-4 significantly outperforms CLAUDE-V V1 I at generating 25 fully- functional devices from the MICRO25 benchmark, reaching near pear-perfect performance. When used as a design assistant for generating six more complex 2022; Liu et al., 2022a), and leveraging these abil- ities may enable new applications. For example, in the context of assistive devices, GPT-4 is able to infer that an ultrasonic sensor can be used to create assistive glasses to aid the visually impaired with navigation. Similarly, the model can use its common-sense knowledge to design devices that contain the most common emojis, or keys for all the prime numbers up to 20 just as easily. Ultimately, electronic devices may be distributed as templates, that can be semi-autommn aumimiiltmati customized to va- riety of applications based on user preferences.
Generating devices with common-sense knowl- edge: Language models contain a variety of common-sense reasoning abilities (West et al.,
devices, language models can generate devices that nearly meet specifications, but still require moder- ate correction by domain experts function. While this novel application of language models inspires the democratization of electronic device creation, further development is currently tempered by the lack of simulators to automatically evaluate de- signs, and the highly manual nature of this process.
8 Limitations
This work has a number of limitations, including:
Device scope: The devices generated in this work are small in scope, with limited functionality -typi- cally small number of components, fewer than 50 lines of code, and controlled by ARDUINO micro- controllers which are frequently limited to only 2K of memory. This work does not address designing moderate or complex devices such as phones, per- sonal computers, or other devices that are orders of magnitude more complex in terms of component counts and code length. For context, being able successfully design all the devices in the MICRO25 benehmark would be equivalent to the performance of a particularly strong undergraduate student af- ter having taken a first course in microcontroller design at our institution.
Quantifying time savings of automatic versus human device design: Precisely quantifying the benefits of automated coding assistants (such as GITHUB COPILOT) is challenging, and currently measured least in-part with qualitative measures (Ziegler al., 2022). These assistants may provide large time savings when they function correctly, but likely increase debugging time when they gener- ate problematic code, complicating measuring their precise benefit. A similar situation likely exists here for device generation, and we provide only the following anecdotal account: the best-performing GPT-4 model described in this work produces in minutes what undergraduates in our course might initially take hours to days to perform, as they learn to adapt their existing computer science skills to the electronics and microcontroller domain. As such, in the tear- systems such as WORDS2WIRES might be viewed as productivity assistants that al- low (for example) scientists with existing coding skills but minimal electronics knowledge to quickly design instrumentation (such as data loggers) or other customized devices that are relatively mod- est in scope and complexity with a minimal time investment.
While the simple devices in Experiment 2 can reach high generation accuracy, particularly with GPT-4, nearly all devices in the more complex open generation condition in Experi- ment 3 contained errors, and required correction by domain expert. In the open generation, three of six devices (emoji keyboard, non-contact tempera- ture sensor, ultrasonic glasses) were generated in essentially functional forms in their base conditions (i.e. before adding additional requirements, such as
Generation accuracy:

---

playing music when keys are pressed), A detailed error analysis is provided in APPENDIX D.
tants may similarly increase productivity in elec- tronic device design, reducing the design process from days to hours (or, minutes). Currently, lan- guage models make a variety of errors on complex devices, and these errors are not always easy to predict. As such, the utility of language models as design assistants may be tempered in the near-term by the time required to manually review every as- pect of design for accuracy. As simulators and other automated evaluation methods become avail- able, some of this burden of manual design review will decrease.
The physical manufacturing of the devices including building circuits on prototyping breadboards using jumper wires, designing printed circuit boards, or design- ing physical 3D printed or laser cut enclosures, was entirely manual and completed by a human. While technologics (such autorouting) exist to auto- mate some of these aspects, they were not used in this work. Similarly, while language models have been shown to have some capacity for generating 3D object models (c.g. Nichol et al., 2022), that capacity has not yet developed to where it would be possible to generate enclosures or other mounting hardware required for physical device construction.
Physical design and manufacture:
Acknowledgements
We thank Peter Clark, Ashish Sabharwal, and the 4 anonymous reviewers for helpful comments on this work, well the Allen Institute of Artificial Intelligence (A12) for funding this work.
Safety: Constructing electronic devices has real dangers and potential harms, including but not lim- ited to the risk of fire, electrical shock, and equip- ment damage, and should not be attempted by non- experts. The development environment is notori- ously hostile to components, and even experienced electrical engineers frequently face safety chal- lenges or accidentally destroy components. Gener- ated devices should always be vetted by a domain expert, and not used for safety-critica applications, upplications where harmful unintended effects may be possible.
References
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoc Le, et 2021. Program synthesis with large language models. arXiv preprint arXiv: arXiv:2108.0773:: 178.0773..
Massimo Banzi and Michael Shiloh. 2022. Getting started with Arduino. Maker Media, Inc.
Jason Blocklove, Siddharth Garg, Ramesh Karri, and Hammond Pearce. 2023. Chip-chat: Challenges and opportunities in conversational hardware design. arXiv preprint arXiv:2305.1324.
Scope of Component Knowledge: Popular elec- tronics distributors in the US currently stock mil- lions of different electronic components. Though many of those components belong to particular component classes that largely share pinout in- formation (for example, DIGIKEY, a popular US- based distributor, lists approximately 1.5 million specific resistors, each with two pins), many of these such the 194,000 sensors currently avail- able do not generally share common pinouts or functions. The 100 common electronic components used in the PINS 1 00 benchmark are representative of common components found on electronics tu- torial websites, and that are frequently required to cir-
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint wxXiv:2177003374
Daniel Shephhd and Alán Aspuru-Guzik, 2023. Language models can generate molecules, materi- als, and protein binding sites directly in three di- mensions as xyz, cif, and pdb files. arXiv preprint 2305.000500500
Daniel Fried, Armen Aghajanyan, Jessy Lin. Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. 2023. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learn- ing Representations.
build basic digital picrocontroller-cootroo/oo cuits but are by no means an exhaustive set of the possible components available to construct elec-
tronic circuits.
Speeding electronic design: Just as coding as- sistants such as GITHUR CoPILOT can increase human productivity for coding tasks (Ziegler et al., 2022), the use of a suite of electronic design assis-
Dan Hendrycks, Steven Basart, Saurav Kadavath, Man- tas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring coding challenge com- petence with apps. NewrIPS.

---

Guyue Huang, Jingbo Hu, Yifan He, Jialong Liu, Mingyuan Ma, Zhaoyang Shen, Juejian Wu, Yuanfan Xu, Hengrui Zhang. Kai Zhong, et al. 2021. Ma- chine learning for electronic design automation: survey. ACM Transactions on Design Automation of Electronic Systems (TODAES), 26(5):1-46.
Zhichao Liu, Ruth A Roberts, Madhu Lal-Nag, Xi Chen. Ruili Huang, and Weida Tong. 2021. Ai-based lan- guage models powering drug discovery and develop- ment. Drug Discovery Today, 26(1 1):2593-2607.
Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang. and Graham Neubig. 2022. Language models of code are few-shot commonsense learners. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 1384-1403, Abu Dhabi, United Arah Emirates. Association for Com- putational Linguistics.
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee. Oded Padon, Alex Aiken, and Percy S Liang. 2019. Spoc: Search-based pseudocode to code. Ad- vances in Neural Information Processing Systems, 32.
Yuhang Lai, Chengxi Li. Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke 7eulemoyer, Scott Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2022. Ds- 1000: natural and reliable benchmark for data science code generation. arXiv preprint vrXiv:2211.11500
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. 2022. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXi:
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang. Yingho Zhou, Silvio Savarese, and Caiming Xiong. 2023. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff. Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li. Jenny Chim, Qian Liu, Evgenii Zheltomozhskii, Terry Yue Zhuo, Thomas Wang. Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko. Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov. Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey. Zhihan Zhang. Nour Fahmny, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Gavin,, Dan- ish Contractor, Siva Reddy. Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Scan Hughes, Thomas Wolf, Arjun Guba, Leandro von Werra, and Harm de Vries. 2023. Starcoder: may the source be with you!
OpenAl. 2023. Gpt-4 technical report.
Aditya Ramesh, Prafulla Dhariwal. Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06/25.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal rein- forcement learning.
Shailja Thakur, Baleegh Ahmad, Zhenxing Fan, Ham- mond Pearce, Benjamin Tan, Ramesh Karri, Brendan Dolan-Gavitt, and Siddharth Garg. 2023. Bench- marking large language models for automated ver- ilog ril code generation. In 2023 Design, Automation & Test in Europe Conference & Exhibition (DATE). pages 1-6. IEEE
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schritwieser. Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, al. 2022. Competition-level code generation with alphacode. Science, 378(6624):1092 1097.
Yue Wang. Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. 2023. Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.0799 077222..
Jiacheng Liu, Skyler Hallinan, Ximing Lu, Pengfei He. Sean Welleck, Hannanch Hajishirzi, and Yejin Choi. 2022a. Rainier: Reinforced knowledge introspector Proceed-
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi. Quoc V Le. and Denny Zhou. 2022. Chain-of-thought prompt- ing clicits reasoming in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824-24837, 24837. Curran Associates, Inc.
for commonsense question answering. ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8938-899 9938-8958, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Zejie Liu, Xiaoyu Hu, Deyu Zhou, Lin Li, Xu Zhang, and Yanzheng Xiang, 2022b. Code generation from floweharts with texts: A benchmark dataset and an approach. In Findings of the Association for Com- putational Linguistics: EMNLP 2022, pages 6069- 6077, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu. Sean Welleck, and Yejin Choi. 2022. Symbolic knowledge distillation: from general language mod- els to commonsense models. In Proceedings of the 2022 Conference of the North American Chapter of

---

the Association for Computational Linguistics: Hu- man Language Technologies, pages 4602-4625 Seat- tle. United States. Association for Computational Linguistics.
different pin names than an official datasheet, and these pins need to be manually aligned by a domain expert by searching through variety of reference materials.
Albert Ziegler, Eirini Kalliamvakou, X Alice Li, An- drew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. 2022. Pro- ductivity assessment of neural code completion. In Proceedings of the 6th ACM SIGPLAN International Symposium Machine Programming, pages 21-29.
B.1.1 Example of a non-critical pin
The permissive scoring metric for the PINS 100 benchmark allows for missing or incorrect infor- mation for non-critical pins. A non-critical pin in this context is defined as a pin that not, strictly speaking, required for basic use of a component. An example of non-critical pin is the DRDY (data-ready) pin on an HMC5883L magnetometer.
A Experiment Hyperparameters
Models (GPT3.5, GPT-4, CLAUDE-VI V1) used sim- ilar hyperparameters through all experiments, with precise configurations and APIs available on the GITHUB repository. Models used greedy decod- ing (temperature = 0) to make experiments near- deterministic, though due to hardware-level imple- mentation and other issues abstracted away though vendor APIS, model output still can change be- tween successive runs, As such, specific (cached) model output for each experiment is provided in the GITHUB repository.
The magnetometer is an 12C device that nominally requires only 4 wires function (if wired breadboard): SDA, SCL,, VDD, and GND. The DRDY pin is marked as optional (not required) on the datasheet, as connecting the DRDY pin primar- ily allows faster polling rates by signifying when measurements are ready to be read by the host mi- crocontroller.
B.2 MICRO25 Benchmark Evaluation
B Additional Experiment Details
Automatic evaluation of full devices faces num- ber of challenges, including that many different solutions are possible for each task, and existing simulators typically lack many of the possible com- ponents a model might generate in a solution. As such, evaluation was conducted manually by do- main expert. The schematics and code were manu- ally inspected for functionality. When non- con-trivial or uncommon solutions were generated, the cireuits were evaluated by constructing them in simula- tor (AUTODESK TINKERCAD, shown in Figure 3) when possible, or physically building the circuits when not possible. When circuits used difficult to source or obsolete components, evaluation oc- curred through manual inspection, and comparing the generated schematics and code to reference ma- terials. For schematic to be considered correct, it must contain all relevant components, and be wired correctly way where code could be written accomplish the desired task. For code to be con- sidered correct, it must correctly perform the task given the generated schematic or for cases where the schematic was incorrect, be able to accomplish the task as-is were the schematic corrected.
B.1 PINS100 Benchmark Evaluation
Two additional considerations complicate evalua- tion, which are addressed here. First, the same electrical component may come in different pack- ages, be built by different manufacturers for example, the power pin (e.g. VcC) on a given com- ponent might be on pin in one package, and pin on a different package. As such, evaluation requires only that the pin name be correct (c.g. "Vcc"), but does not require producing pin numbers, which are typically matched when choosing a particular component package during the circuit board de- sign phase. Second, the specific names for pins are often described differently and frequently with only single letters. For example, pin with re- set functionality might be described variously as "RESET", "Rst", or even simply "R" in different sources of text, such as official part datasheets or web tutorials, with each label being correct. As such, model-generated output is evaluated fully manually by domain expert, who requires either generated pin names to match official documenta- tion, obvious short forms, (in the case of large differences) alternative part naming conventions found through a web search. This alignment pro- cess mirrors the actual electronics design process, where a reference schematic for a circuit may use
B.3 Description of Human Evaluator
The domain expert used for evaluating this work is an author this work, with the following qualifi- cations: The evaluator is an award- wainningnnng science

---

educator, and prolific internationallyxecoonizedddd open source hardware author with approximately 50 articles describing their open source hardware work in popular international news media such as Reuters, Forbes, and the Washington Post. The benchmarks described in this work were both au- thored and evaluated by the domain expert reflec- tive of the content of popular full-term (4 month) undergraduate course in rapid prototyping and mi- crocontroller design intended for computer science and information students, typically undertaken in a student's s final year of undergraduate studics at an RI ("very high research activity") university in the United States. The domain expert has delivered this course approximately 10 times to approximately 500 undergraduate students.
2. Emoji keyboard: a USB keyboard that only contains keys for common emoji characters.
Spectrometer:
visible light spectrometer using the uncommon Hamamatsu microspec- trometer, and that displays the spectrum on an organic LED (OLED) display.
Non-contact temperature: a device that mea- sures the temperature of an object using com- mon infrared based non-contact temperature sensor, but displays the temperature in an un- common way: color-changing bar graph an LED display.
In addition, two assistive devices were explored:
B.4 Representation of Full Devices
5.
Pill alarm: common pill-alarm, that dis- plays the current time an LCD display. The alarm is presented in uncommon way: by using servo motors to physically waive flags that say "take pill X" for users, until they press a resct button.
Additional information on device representation in Figure 3: The bill of materials format expresses canonical information typical in the design pro- cess, including the component type (e.g. "resistor"), component name in the schematic (e.g. "R1"), com- ponent value (e.g. "10k ohms"), well as a note on the purpose of the component (e.g. "current limiting resistor for LED"). The pinouts are ex- pressed as a dictionary containing lists of pins for each part, as in Experiment Code is expressed between MARKDOWN code blocks to ease extrac- tion. Schematics are expressed as "netlists", which are common storage format frequently adapted by electronic design tools. This format is analogous to undirected graph, where edges represent given connection from one component pin (such as the anode of an LED) to another component pin (such as one terminal current-limine limmtini resistor).
Ultrasonic glasses: common components (an ultrasonic distance sensor and piczo buzzer) used for an uncommon purpose to create a pair of glasses for the visually-impaire The glasses audibly notify the user of the distance objects in front of them using tone whose frequency varies with distance.
C Prompts
The full prompt for Experiment (component pinouts) is provided in Table 3, while the full prompt for Experiments 2 and 3 (device genera- tion) is provided in Table 6. All prompts are static that is, the same format/n-shot examples shown here are also shown in every generation request with the exception of the task strings ([bolded) the tables), which are substituted with task-specific strings (representing the specific user-requested de- vice to generate) at runtime. For Experiment I, this is limited to the component name to generate pinout information for (e.g. "DC motor"). For experiments 2 and 3, this is limited the target microcontroller platform (e.g. "Arduino Uno") and plain-text device description (e.g. "create a USB keyboard that only has buttons for the 9 most popular emojts on it".
B.5 Description of Open-Generation Devices Six devices, crafted to use uncommon components, common components in uncommon ways. The six devices are:
1. Random number generator: a random num- ber generator, using two uncommon compo- nents: (a) radiation sensor to help provide random seed based on ambient radiation levels, and (b) high-voltage nixic tube (or cold-cathode display), similar vacuum tube, to display randomly generated digits. Nixie tubes were manufactured and used in the 1950s and 1960s before light-emiting diodes became common.

---

Your task to generate description and pinout for electranic compunent. specific electronic corponent to generate this output for is: (componentNane) The output fornat is JSON, between code blocks, as shown in the exaple belaw:
"7479":
"deseription"; "Danl positive-edge triggered flip flop, asynchronaus presst and clear", "D"R""t:"""M"I", "DI", "CLKI", "MPRI", "QI", "VQI", "VSS", "#Q2". "Q2", "APR2", "CLK2", "D2", "AR2". "VD0"]
Table 3: The prompt for Experiment (component pinout generation) on the PINS benchmark. The specific 1-shot example in the format prompt (the the 7479 FLIP FLOP) is used across every experimental run only the string representing the target device to generate pinout information (ccomponenNName), egg. "DC MOTOR") changes.
Error Analysis: Modifications to Open Generation Devices
needed to be clocked out the spectrometer and read by an analog-to-digital converter after sending start pulse to the spectrometer - but the generated code had incorrect clock timings, logic levels, and other fine-graine' details which made it unable to function without correction.
D.1 High-level Qualitative Challenges
Overall high-level qualitative challenges in design- ing the six case study devices are described briefly below, where a detailed description of errors and corrections for each of the six devices is provided in APPENDIX D.2.
Deprecated or mismatched libraries: Hardwarc libraries are frequently updated to support new fea- tures, but the ARDUINO ecosystem lacks Makefiles or build manager with explicit library version numbering. making specific library versions un- known for given code example. During manu- facture, this frequently required searching through old versions of library (such as the Bounce but- ton input library for the emoji keyboard) to find a version that matched the specific API generated by the model. Similarly, observing examples of dif- ferent APIs across different library versions causes the model to occasionally mix APIs from older and newer versions of libraries, from different libraries with similar functions (such combining the APIs of several LCD display libraries for the spectrometer).
Sensitivity to prompt: Small and seemingly help- ful changes in the device description prompt can cause large changes in generation. For example, for the random number generator, including re- minder that the radiation sensor required pull-up resistor appeared to cause the model to forget to include a high-voltage supply for the nixie tube.
Many requirements can create poor perfor- mance: Adding many composite requirements to a project, even when they are individually easy, can create low performance. For example, for the emoji keyboard, adding the requirement to play relevant musical tune when each key is pressed generally produced only scaffolds for music gener- ation code without actually including the melodies, A subsequent call GPT-4 asking it only fill this music scaffold was required to generate the melodies.
End-of-lift parts: Electronic components regu- larly reach end-of-life cycles, and are no longer manufactured or easily available. The model oc- casionally generated circuits that used unavailable parts, and had much less competency generating circuits for newer part variants, particularly those that were released near GPT-4's knowledge cutoff date of September 2021 (OpenAI, 2023).
Device drivers: The model performs best for straight-forward circuits where the coding portion of interfacing with external components (such as sensors) is abstracted to existing libraries. When writing low-level device library is required, the model commonly either hallucinates non-cxistent library, or generates a reasonable first-pass de- vice library that requires extensive modification function. For example, the microspectrometer device driver the model generated had the essen- tial lenceptual-level components - i.e. that data
D.2 Device-Specific Errors
The devices generated in the open generation exper- iment generally required modification function as intended. Here we provide a list the major design changes required to reach functionality:

---

D.2.1 Random number generator
using a library for common component (a dis- play with a common controller). Using different device descriptions, the model either hallucinated non-existent libraries for interfacing to the spec- trometer, or generated its own libraries that had the high-level procedure correct (c.g. sending a start pulse to the spectrometer, then continuously sending clock pulse while reading data using analog-to-digital converter read out each of the 256 spectral channels) though the specifics of the device driver, such as timing or logic levels were typically incorrect and needed to be manu- ally corrected. With respect to the display, three OLED and TFT displays with common display con- trollers were attempted, and the most successful (the 128x 18x1228 128 OLED using SSD1351 controller) was used. There were only two small errors in the display code: the initial call to the display had re- versed the order of the arguments, and the last call to the display (swapping the backbuffer) was for a different library, and not required here.
Both the radiation sensor (Radiation Watch Type 5) and the nixie tubes (IN-/2A) are highly uncommon components, and are likely to have limited exam- ples available in existing documentation. Gener- ally, across several iterations of device description prompts, either the radiation sensing circuit was correct, or the nixie tube circuit was correct, but not both. The radiation detector requires pull-up resistor to function, and is pulled low when high- energy particle strikes it. The nixie tube requires an external high-voltage driver, which was usually generated correctly, but when generated incorrectly it was typically powered by USB voltage (5V) in- stead of the required high-voltage (170V). Across device descriptions, the method used in the code to set the random seed based on the radiation sensor varied some useful, some largely incorrect.
D.2.2 Emoji keyboard
The base emoji keyboard was largely generated without issue, though did mix up the version for the button input library, and failed to mention the spe- cial programming requirements for the Teensy mi- crocontroller to place it in human-interaace-evicc (HID) mode to act as USB keyboard. The specific emojis were chosen by the model, adding only the requirement that they must be high-froee tigh-frequency, and at least must be the heart emoji. Adding many requirements generally reduced design quality - for example, adding the requirement that some of the emojis needed to be at least 5 ASCII characters long was not generally successful (and, the model occasionally generated emojis that were unicode. which is generally easily supported by the USB HID standard, occasionally generated only sin- gle characters instead of full emojis). Similarly, adding the requirement for short musical tune play upon pressing an emoji, where the tune should have a similar affect to the emoji (e.g. a love song for the heart emoji, a happy song for the happy emoji, etc.) generally produced only harsh single tones, or the scaffold for generating the mu- sie without actual musical tones for each emoji. This scaffold provided to GPT-4 on its own in
D.4 Non-contact temperature sensor
This device consisted of two common components (an MLX90614 non-contact temperature sensor, and an 8-pixel neopixel RGB LED strip). The device generated without issuc, and only mod- ified slightly to reverse the direction the LED bar graph displayed from (to accommodate the mount- ing constraints of the specific LED strip used).
D.5 Pill alarm assistive device
This device consisted of three common compo- nents: an 16x2 LCD with an 12C interface, three hobby servos, and single pushbutton. The schematics generated largely without issue. The code generally had number of logic errors that needed correction when the added requirement of oscillating the flags back-and-forte was added, in- cluding that the code would oscillate all flags, re- gardless of which alarm (e.g. pill 1, pill 2, or pill 3) was active. Different generated instances of this device in response to different specifications either kept track of time internally, or used an external realtime clock module for more accurate timekeep- ing but all generated devices failed to provide any means of setting the initial time of the device other than manually in code, which an important usability feature a clock not explicilly mentioned in the prompt.
step, and the resultant code added to the original code.
D.3
Visible Spectrometer
This code had two central challenges: generating a device driver for an uncommon component (the Hamamatsu C12666MA micro-spectromeeter,, and

---

Calegory
Tusk Description String
Digital Button
Create device with a single push button. on pin when button pressed, and turns off that same ID pin when
the button is not pressed.
Digiuu Mulliple
Create device with 4 puslt buttons, that turns L/O pin when exactly of the buttons butlon are pressed, and turns off that
1/ pin otherwise.
Analog
Create device that reads the analog from potentioencter configured as a voltage divider and soureed by 5V. When the input is greater or equal to 2.5V, it should tumn an UO pin, while the inpur helow 2.5V should off the IO pin.
Protoculs
Soria PURTT
Create a device that reads the Serial port at 9600 band. Whenever the string 'hello" transmitted to the device, it will respond by sending "world".
Seria//PI
Create a device that connects to another device using SPI Whenever the string "belle" is transmitted to the device, it will respond hy sending "worl"".
12C
Create device that reads value an 12C device, displays value the serial pon every second. The device address
is 0x50, the register to read is 0k15, and the value is bins long. The value should be displayed in base 10.
Card
Create device that opens a file called "outaxi" SD card, and and every 10 seconds, randomly prints one of following animal names (as well as newline character): cat, dog, mouse, parott
Outpul
Motor DC
Create device that oscillates between spinning DC moter one dircetion then the other direction every 5 secoeds.
Motor Stepper
Create device that occillates the output a stepper moter clockwise counterclackwise 45 degrees, every 5 seconds.
Motor Servo
Create a device that contiuously moves a lubby servo back and forth from 45 degrees to 135 degrees every 5 seconds.
LED Blink
Create device that blinks LED every 500 milliscconds.
LED Sequence
Create a device with 4 LEDs, that blink sequence. one after the other, every 500 millisecomds. When the end of the sequence is reached, the pattern should reses, and continue indefinitely.
LED 7 Segment
Create a device that counts from to 9 on a segment display, indexing numbers every 500 millisccoeds. When the end of the eycle is reached, chould start again.
LED Neopixel
Create device with a neopixel (WS2812) that skowly continouuly eycles rainbow colors.
Roley
Create a devioe that turns a relay on for 2 scconds, then off for 5 seconds, continuously. The relay coil takes 500 milliamps of current to engage.
LCD
Create a device that prints the plaas "Hello World' on 16x2 LCD. Use a IID44780 compuibble 16x2 LCD, configured normally
(l.c. withour an I2C, Serial, or other simpler connection).
Sound Buzzer
Create device that continausly plays a high. medium.
on a Piezo Buzzer, changing tones every second. Create device that produces sawtooth wave, ramping up from 0V to 5V every 50 milliseconds.
Analog Ourput
Sensors
Resistive -ipiive---SS CDS
Create device that reads the value from CDS cell, and ourputs it to the Sorial port every socoad.
Manual Protocol
Create a device that reads the distance from a IIC SR04 ultrasonic distance sensor, and ousputs the distance (in centimeters) to the
Serial port every second.
12C Magnetic
Create a device that reads current magnetic field readings using HMC58K1, magnetometer. The readings (x, y, field
strengch) should be uuiput to the Serial part every second.
Logle
Simon
Create device that implements the popular memory game simon, where enter progressively longer sequences of colors. It should have 4 possible colors, and include sound when the button is touched, as well as when winning/lssing. The game should timevut the user doesn't enter input after 5 secands.
Conway
Create device that implements
popular Conway's Game of Life, on 8x8 LED matris. The game steps should cycle every 500 milliscondss If the board is empty. it should randocily initialize the game again. There should be pushbutton that allows randomly resetling the gume.
Clock
Create a device that implements a clock that prints the current time on a 16x2 character LCD display with on 12C interface. should frieldly
have ihree butions to help set the time
increment hours, soconds. The tinckoeping should be performed by the Arduino, and not an external real-time clock.
increment minutes, and one increment
Air Temperature
Create a device that reads the current air temperature, and displays it as a color on a neopixel (5) The color should he fally blue OC, fully red 30C, and an interpolation of blue and red between Uhose lemperalures.
Table 4: The full (i.e. not truncated or summarized) task strings used for the MICRO25 benchmark. The target platform for each is the ARDUINO UNO microcontroller.
D.6 Ultrasonic glasses assistive device
generated by the model was also modified to a re- duced range and more fitting for human ears, as the original included figh-freqeency tones that, while audible, were uncomfortable and resembled a fire alarm.
This device generated with only small issues. The device requirements specified using a specific battery-powered ESP32 microcontroller board, but the schematic used digital pin numbers that were unavailable on this specific microcontroller board these were assigned to other pins trivially. The li- brary the model used for sound generation (TONE) is famously available for most Arduino devices except the ESP32, and was modified to use dif- ferent function specific to the ESP32 with a similar signature, but with two added initialization and ter- mination calls. The specific audio frequency range
E Device Descriptions: 25 Benchmark Tasks and 6 Open-generation Devices
The full device task description strings for the MI- CRO25 benchmark are shown in Table E. A set of iterated task description strings for the open- generation condition (Experiment 3) are provided in Table 5. These and additional task descriptions are provided in the GiTHUB repository.

---

Device
Task Deseription String
Random number generaior
Radiouctive device that uses radiation rate radiation watch type
(which outputs digital signal. active low, depending on whether high-cnergy particle has struck it at that moment or not) to determine the maddo sood electronic dice. The device should continualyy read the radiation change the random seed periodically. Every seconds. the device should display the roll of a sided dice on Nixie tube. It should use an IN-12 nixic rabe, and K KI5SIDI driver.
accumulate the count, and
help
Emaji beybuard
Create a keyboard that plugs
keyboard has only small number buttons. The keybourd should only have buttions for 9 popular emejis, expressed as ASCU characters, not unicode. One emoii should be LED
device.
beart.
should
the time. blinks off for milliscconds when button is pressed.
There shauld also he a pieza buzzer, that plays a hrief mane that is of the same affect the emoji being pressed
for exumple. love song for the heart emoji. happy song for huppy emoji. sad music for sad emoji. and so forth. Create a visiblc spectrometer
Visible light speetrometer
contimously displays the spectrum on OLED display. It should use the Hamamatsu
CI 26666M Sv.compatible mini spectrometer for the spectrometer (pins: SV, GND, EOS, START, CLK, GAIN VIDEO). computbble (pins:
pisel OLED GND, VIN, CD. MISO SICS, OL EDCS, RESET, DC. SCK, MOSI).
display should be
SSD1351 controller SPI interface,
Non-contact temporature sensor
non-contact temperature sensor strip. D degrees or below should light only the first neopixel. For each IDC after, another nespixel should light. The color of the neopixel should change according its temperature (bluescold, green=mild, yellowswarm. orange-warmer, redwhot).
temperature should outpur 8-pixel ncopixel
Pill alarm
Create a pill alarm. The alarm should have a clock that prints the current time on a 16x2 character LCD display. If the Blags (signifying different pills to taken).
time 6:30am, noon, 6:30pm, device should Serva controls flag I, servo controls flag 2, and servo controls flag 3. When raised, the servo should move from degrees 90 degrees. The should stay up until button pressed, after which it's
the down position (D degrees). While raised, the servos should slowly oscillate between 45 and degrees, to help get the user's attention. Glasses for the blind that provide helpful sound that corresponds to how close something in front of them. Should have sound. Please the MaxSonar ultrasonic disrance sensor.
Ultrasonic Glasses
switch that
Table Example iterated task deseription strings for the 6 open-generation devices in Experiment 3. A of initial task descriptions was progressively iterated, expanded, and refined based on task performance, before arriving the above task descriptions.

---

You DeveloperGPT, the advanced Al developer tool planet. You any coding question, and provide real useful exarple code using code blocks. Even when you are not faniliar with the answer, you use your extreme Intelligence figure it out.
ther, you have specialized training electronics,
design enbedded electronic circuits wucontoollePlaafoonsse platforn, coupled with prograns to make those circuits successfully accomplish tacks.
Your task is (taskStr)
generata following:
A bill of naterials, in JSON form (see fornat belos).
parts, key being part name, and the valun baing a 1ist of all pins the part has, to halp in gonerating the schematic. À schenatic, 1SON form (see fornat below). Each line of the scheratic should describe single connection in the circuit. pregran that implerents the progran to successfully complete the task. Each section should between code blocks special instructions, point form, required.
pinout,
below). pinout
dictionary
A complete mnieroontrollerlaffoone
brief
aoditional reninders:
there possible, a descriation/part number of the device should be included in the nates. Alternatively, where nany parts could be substituted, it should include criticel information to nake that choice (such the controller required for an LCD displey, or the voltage required for an LED) The coda should be complete. It can #include built-in (aleroccntrrlllrrttaatoonsss libraries, but otherwise code corpile the Arduino Uno platfors.
as-is.
Hore exarpla output for genarating device that blinks LEDs alltrratting pättern every secand,
Bill naterials:
("part":"Arouing "Arouino Uno", "nare":"und" "value";",, "" "notes": "Anduino Uno mtcrocontrollerr),
("part":"LED", "Dane"-"""", "value"i"ret "notes": "odtes":'alten "slternating LED Standard valtage (2-3.3V), ("pert":"LED" "name" "D2", "value" "aalue":"hitt"" :"hite", "notes"; "allermatnn LED Standard voltage range (2-3. (233.3)). "). "Raststor", "Rasstt""""ssssorr,,, "nare"""""" "RI", 1228 ohr", "curratt linieing reststor for LED1 at 5V"), ("part"; "Reststor", "nare" "R2", "R2",
"value":"22 1228 ohr", "notas": "current liniting reststor LE02 at 5V"),
1
Pinouts:
"Anduino Uno": [*sy", "3.3V", "OND", "AREF" "0a/Rx", "01/Tx", "02", "03", "D4", "D5", "08", '07", "DS", 109", "010", "D12", "Di3", "A8", "Al", "A2", "A3", "M4/SDW" "AS/SCL"], "D1": ["anude", "cathode"],
'02": ["anode", "cathode"],
'81" ['1", '2"],
'82" ["1", '2]
}
Schematic (List of connections):
..
[
"D"" "pin": "cathode"" ("nane"; "uno", "GND"),, # Connect D1 cathode Uno GND
[("hame":"ur", "pan"s"anode"" ("hare": "R1", "pin" (2")), Connect
pin [("nane"="R1" "1"), ("nane": "und", "pin"="DS")J. Connect pin of RI (current liniting resistor) to Digital
(current liniting resistor)
activate/deactivate "D"" "cathoeee, ("nane"; "uno", "pin""""ND""]] # Connect D2 cathode to Uho GND
[("hane"=02*% "anode",, ("hare": "Rz", "min" "2")), Connect anode pin R2 (current limiting resistor)
Connect pin of R2 (current liniting resistor) to [["hane": "RR2" "pin":"1"), ("hane": "uno", "pin":"D""]]. Uno Digital I/C (D6), activata/deecctivate 02
1
Arduino Uno Code:
Alternating blink
// This code interfaces with circuit that has two LEDS that blink in alternating pattern.
The pattern changes every second.
// LED Digital 1/0
adefine PIN_LED1 5
// LED Digital I/0
sdafine PIN_IED2
setup wold satup()
once you press reset power
// Initialize LED pins to output node Pin OUTPUT):
LE1, OUTPUT);
Loop (Pront continues anto next page...)
again

---

(Pronpt continued fron provious page page...) void 1oep()
digttallrItee(NN.EE. HIGH); // Turn LED
te LOM); // Turn LED OFF delay(1880);
digttalliiite(PIN.EED. HIGH); // Turn LED OFF
digttallwItt((IN_LEED LOM);
d:lay(1800); // wait for second
Instructions:
This code uses only standard libraries, No additional libraries are required in the library manager. Assasbla cirautt and progran nornal.
Snippet examples (also for the Arduino Uno):
Example: Connecting servo Materials:
[
("part": "Servo Motor", " "name":"S!" "S"", "value":" :"", "notes":" :"Standerd 3-wire SV compatible hobby servo (e.p. SC98)")
]
Pincuts:
Arduino anitted for snippet
"Servo Motor": ("VCC", "GND", "signal")
Schenatic (List of connections): ...
[
"S"han""="1"", "signal",,, ("name:: "une", (o3")), * Connact Servo signal to Uno na [("hane"="51", "VEC"), ("nane"; "uno", "p""" Connect Servo Uno [("nae":""s"" "GND", ("nane"; "uno", "pin":"GND""" Connect Serva SMD Una GND
]
Exarple: Connecting a button (oull-up) Bill of Haterials:
...
[
("part": "Button", "nane": "value":">, "notes": ""ooenttayy button"),
("part": "Resistor", "nane" :RR",, "value":"io "iok ohr", "notes" "Pullluup resistor for button")
Pinouts:
...
Arduiino anitted for snippet. "Button": D'1", '2"J,
"Resistor": ["1", "2*]
}
Schematic conmections):
[
(3"""ne"""T""", "pin":"l"), C"nane:: Tune", "0""" Connect Button pin Uno D2 [("aanet""m"" "pin":"""), ("nane"; "RI", "p"n"=""")], Connect Button pin to BT pin [("hane":""1", "pin": "2"", ("hare": "una", "pin":"5v""1, Connect pän
(pull-up)
"("nane":""T1", "2"):"""), ('nare": "uno", "oin"">"xD">> )] Connect Button pin 2 to GND
].
Example: This is case what NOT do. Schenatic (list of connections): ...
[C"nane": "ICI", "pin": "Irputs"), ("nane": "uno", "pin": "D5-D18")] * is not clear which pin on the is connected to which pin on the Una.
individuslly.
Please generate the b:ll of noterials, pinouts, scheratic, code, and any special instructions for the requested task below. should commented, The platforn is: (stercontrlllePlaatoonns)) logic, prevent any bugs.
The task is: (taskStr).
Table 6: The full prompt for device generation, which includes one full positive device example (a simple device that blinks two LEDs in alternating pottern), well as two positive snippets (illustrating component with more than pins (servo), and a connection with more than 2 components (pull-up resistor), respectively), and one negative snippet designed promote explicitly enumerating schematic connections. At runtime, pnierocontrollerrlaffooom Arduino Uno), and (taskStr) is replaced with the description of the target device generate (e.g. from Tables 4 or 5).
replaced with the platform (c.g.


Reviews:
[{"review_id": "vDqJyguuIU", "reviewer": "Reviewer_P9mH", "summary": "", "questions": "* Question A: You say that the LLMs are instruction fine-tuned. What kind of instructions did you use and how?\n* Question B: Are there any procedures for what happens when the LLM suggests a part that is not available?\n* Question C: You said that the devices were synthesized by a domain expert that had to correct some common errors. You also state that the LLM's capabilities are comparable to a student from an undergraduate course in microelectronics. Does the LLM reach undergraduate capabilities with or without the corrections? I am still to get a better understanding of the true capabilities of the LLMs.\n* Question D: Can you provide the prompts that you used to design the six real world use cases?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and thoughtful review. \n\nSummary of rebuttal in the context of reasons to reject:\n\n1. **Code design relevant to electronic circuits:** While it is true that our Related Work does not include a detailed discussion of code generation applied to electronic circuits specifically, that is because (to the best of our knowledge) this paper is one of the first works in this new subfield.  ACL policy ( https://www.aclweb.org/adminwiki/index.php/ACL_Policies_for_Submission,_Review_and_Citation ) defines contemporaneous works as those within a 3 month period before the submission window (i.e. after March 2023). We will of course update our related work to include all work published in the interim between submission and camera ready deadlines, but *ACL policy forbids this being used as a reason for rejection: \"In such cases, reviewers should point authors to the non-cited work (so that they can discuss it in the camera-ready version) but not penalize the authors for missing the citation.\"*\n\n2. **Expert evaluation as a negative:** *Expert evaluation performed by a human domain expert is the *gold-standard* research methods and evaluation for NLP, not a reason for rejection.*  Proxy metrics (e.g. BLEU/ROUGE in Machine Translation) are used as inexpensive but inaccurate approximates of performance when expert evaluation is too expensive, and developed only when the field has progressed enough to develop and validate such metrics (e.g. even though character-level overlap was used in the original IBM alginment papers (e.g. pg 84 of Brown et al., 1990, \"A statistical approach to language translation\", Computational Linguistics), it wasn't until 14 years later (Lin, WAS 2004, \"Rouge: A package for automatic evaluation of summaries\") that these metrics became studied in detail and popularized).  Other subfields still haven't made that progression -- *for example, no one would reject a Medical NLP paper because the task required expert evaluation by doctors -- similarly, rejecting a paper in a new subfield because it also requires domain expertise (electrical engineering) for evaluation is unfounded.*  ACL policy ( https://2023.aclweb.org/blog/review-acl23/ ) cites that \"Having no precedent in existing literature: Believe it or not: papers that are more novel tend to be harder to publish. Reviewers may be unnecessarily conservative.\" is not grounds for rejection.  \\\n**Additionally:** The experiments performed in this work were done to the highest standards of natural language processing research methods.  The code, benchmarks, library, and experimental results are committed to a github repository, awaiting public release.  \\\n**Objectivity:** While code generation (and, circuit generation) can be notoriously difficult to automatically evaluate because: (a) there are many possible valid code and circuit solutions for a given task, (b) there are no current ways to write unit tests for arbitrary microcontroller hardware (e.g. no simulator supports arbitrary components an LLM might mention, or even more mundane but useful parameters such as reporting what angle a simulated servo is at), the *tasks are objective and straightforward for a domain expert to evaluate*.  Where two doctors might disagree that an elevated liver enzyme is indicative of two different diagnoses, for this task, the measures are completely objective -- i.e. the LED either blinks, or it doesn't; the servo either oscillates back and forth between 45 and 135 degress, or it doesn't; the correct distance is either reported on the serial console for the HC-SR04 ultrasonic distance sensor, or it isn't.  This is essentially the first paper in this subfield, not the last, and future work by our team and others will work to make evaluation faster with more capable simulators (which are a huge amount of infrastructure to stand up; but we're working on it), and in the mean time examining whether new proxy metrics are a viable alternative.\n\n3. **Analysis of time savings vs human effort:** The PI on this project says: \"While we definitely agree it would be nice to have some large human user study on the time savings associated with using large language models for device design, measuring this is not straightforward, and indeed not even yet possible for coding assistants.  For example, even the Github Copilot team (arguably the most-deployed and most-used coding assistant) have difficulty quantifying measurements of productivity just for code assistants ( https://github.blog/2022-09-07-research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/ ), and are currently using survey-based measures (e.g. \"Do you feel more productive when using Github Copilot?\", \"Do you feel you can focus on more satisfying work when using Github Copilot?\", \"Do you feel you use less mental effort on repetitive tasks when using Github Copilot?\").  It would not be possible to solve the research methods problems of quantifying code assitant contributions to productivity (and, running a large user study) in the one additional page afforded by EMNLP, but we look forward to exploring the human-centered computing aspects of this in future work, and submitting those papers to appropriate (e.g. HCI) venues.  What we can likely say from our experience with teaching undergraduates these same concepts is a similar story to Copilot: that for someone without an electronics background, if they wanted to make comparatively simple devices (e.g. a grad student in a non-ECE field wanting to create data loggers for experiments monitoring climate change variables), the system we describe would be invaluable, and enable this to happen at a high confidence without having to take hundreds of hours of coursework.\"\n\n\n**Clarification Response to questions for authors:**\n\n**\"Question A: You say that the LLMs are instruction fine-tuned. What kind of instructions did you use and how?\"**\n*Response:* We will clarify this description in the text.  We use GPT4 and Claude-V1 in this work, which are instruction-tuned models (i.e. they are pretrained on datasets of instructions, to understand user intent and accomplish the task the user describes).  It is not currently possible to fine-tune these models, and all experiments reported in this work are completed using in-context learning with the ~2k token prompt. \n\n**\"Question B: Are there any procedures for what happens when the LLM suggests a part that is not available?\"**\n*Response:* This is a great question.  When the LLM adds a part that is not available, we make every attempt to source it -- for example, the antique IN-12 Nixie Tube (and it's associated K155ID1 driver IC) were procured through dealers in legacy hardware.  An example we didn't include in the paper (but that is included in the repository, for transparency) is an open-generation device that's a magnetic compass that uses a magnetometer combined with a neopixel ring to create a navigational compass.  This device consistently generated with a HMC5883L magnetometer, which was a very popular magnetometer in the open source community that recently reached it's end-of-life.  We tried to produre it, but all the versions that were shipped ended up being unlicenced clone devices (e.g. the QMC5883L) that are not entirely compatible with standard libraries.  This complicates the evaluation (e.g. the code would require modification; but might have worked originally), so for this paper we simply didn't include it.  Ultimately this ends up being analogous to code generation tasks that generate code for libraries that are not only depricated, but also unavailable.  Our proposed solution is described breifly in the discussion (L440-445), where we call on building an open source simulator with an open part library to facilitate automated evaluation, but (also) to constrain design generation to a subset of available components.  While this is useful for end-of-life parts, you can imagine that this type of constraint-based generation might be particularly useful in a few years where a system like Words2Wires might be attached to a pick-and-place machine populated with a library of common parts -- so that you could ask the system to design a device for you and have it made moments later.  In such a case you would need to constrain generation to a subset of parts, and find ways to work with what you have (e.g. substituting components; combining two standard resistor values in series or parallel to make a desired value; etc.). \n\n**\"Question C: You said that the devices were synthesized by a domain expert that had to correct some common errors. You also state that the LLM's capabilities are comparable to a student from an undergraduate course in microelectronics. Does the LLM reach undergraduate capabilities with or without the corrections? I am still to get a better understanding of the true capabilities of the LLMs.\"**\n*Response:* This is a good question, and we will clarify this in the text.  The text (L494-L498) comparing model performance to undergraduate performance is: \"For context, being able to successfully design all the devices in the MICRO25 benchmark would be equivalent to the performance of a particularly strong undergraduate student after having taken a first course in microcontroller design at our institution.\".  For clarity, this means the best model's performance as-is (e.g. uncorrected) on MICRO25 (GPT4, 96%) is equivalent to a strong (95%+ percentile) undergraduate at our institution.  Only the 6 case studies in Experiment 3, where the model is used as an assistant (instead of a PASS@1 generator) had errors corrected. \n\n**\"Question D: Can you provide the prompts that you used to design the six real world use cases?\"**\n*Response:* Yes!  These (and indeed all prompts used for all work in the paper) are provided in the Github repository with a script for replicating them. \n\n*Below are the progression of prompts used to create an ever-more-complex Emoji keyboard.  The model's generations were generally fine up until `keyboard-emoji5`, when the added requirement of playing a musical tune was added -- the model just generated a scaffold (e.g. void playMusic() // add music code here).  Since this was being used as a copilot-style assistant, we just fed that back into the model and asked it to populate that music code, but still count that as an error/omission in the original generation.*\n```\n[keyboard-emoji1] create a keyboard that plugs in as a USB device, but instead of a full keyboard it has only a small number of buttons. The keyboard should only have buttons for 9 popular emojis.  There should be an LED that's on all the time.\n\n[keyboard-emoji2] create a keyboard that plugs in as a USB device, but instead of a full keyboard it has only a small number of buttons. The keyboard should only have buttons for 9 popular emojis, expressed as ASCII characters.  Some should be short, some can be long.  There should be an LED that's on all the time.\n\n[keyboard-emoji3] create a keyboard that plugs in as a USB device, but instead of a full keyboard it has only a small number of buttons. The keyboard should only have buttons for 9 popular emojis, expressed as ASCII characters.  One emoji should be a heart.  One other emoji should be a long string containing at least 5 characters.  There should be an LED that's on all the time, but blinks off for 500 milliseconds when a button is pressed.\n\n[keyboard-emoji4] create a keyboard that plugs in as a USB device, but instead of a full keyboard it has only a small number of buttons. The keyboard should only have buttons for 9 popular emojis, expressed as ASCII characters.  One emoji should be a heart.  One other emoji should be a long string containing at least 5 characters.  There should be an LED that's on all the time, but blinks off for 500 milliseconds when a button is pressed.\n\n[keyboard-emoji5] create a keyboard that plugs in as a USB device, but instead of a full keyboard it has only a small number of buttons. The keyboard should only have buttons for 9 popular emojis, expressed as ASCII characters.  One emoji should be a heart.  One other emoji should be a long string containing at least 5 characters.  There should be an LED that's on all the time, but blinks off for 500 milliseconds when a button is pressed.  There should also be a buzzer, that plays very brief music (i.e. no more than a few seconds) that is of the same affect as the emoji being pressed -- for example, loving for the heart emoji, happy for a happy emoji, sad music for a sad emoji, and so forth.\n\n[keyboard-emoji6] create a keyboard that plugs in as a USB device, but instead of a full keyboard it has only a small number of buttons. The keyboard should only have buttons for 9 popular emojis, expressed as ASCII characters, not unicode.  One emoji should be a heart.  There should be an LED that's on all the time, but blinks off for 500 milliseconds when a button is pressed.  There should also be a buzzer, that plays a brief tune that is of the same affect as the emoji being pressed -- for example, a love song for the heart emoji, a happy song for a happy emoji, sad music for a sad emoji, and so forth.\n\n[keyboard-emoji6a] create a keyboard that plugs in as a USB device, but instead of a full keyboard it has only a small number of buttons. The keyboard should only have buttons for 9 popular emojis, expressed as ASCII characters, not unicode.  One emoji should be a heart.  There should be an LED that's on all the time, but blinks off for 500 milliseconds when a button is pressed.  There should also be a piezo buzzer, that plays a brief tune that is of the same affect as the emoji being pressed -- for example, a love song for the heart emoji, a happy song for a happy emoji, sad music for a sad emoji, and so forth.\n```\n\n**Remarks on reproducibility:** \n\nThis is a work of open source science, and every effort has been made to make the resulting research products easy to use and well-documented for reproducibility.  The code, benchmarks, libraries, and results currently sit in a Github repository awaiting being made public.  Runscripts allow users to regenerate the results in about 12 hours (at current API speeds).  Unlike other tasks that require evaluation by domain experts (e.g. medical tasks, where doctors might disagree that a given symptom is individative of one diagnosis versus another), in our tasks the evaluations are objective -- i.e. the LED either blinks, or it doesn't; the servo motor either oscillates back and forth between 45 and 135 degrees, or it doesn't.  Papers that require (a) domain expertise, or (b) API access are still considered highly replicable (the PIs advisors used to have to travel across the world 20 years ago to access specific Linguistic Data Consortium (LDC) corpora to replicate others work).  Where possible and applicable, the results of this work are augmented with additional information (e.g. screenshots of the simulator, that augment the JSON netlists produced by the model) for usability and to quickly aid in verification.  \n\nThe reproducibility score suggests this work is not reproducible, but we believe this to be in error as every effort has been made to make this work  reflective of *highly reproducible* open source science. \n\n**Missing references:**\n\nThank you for suggesting the additional reference (Thakur et al.) on Verilog code generation, and we will include this in our related work section.  But, it's worth noting that this work (published by IEEE and available online in June 2023 -- i.e. ACL-defined contemporaneous work): (a) Introduces a benchmark of 17 tasks (vs our 25 in Micro25, not including Pins100 or our 6 case studies), (b) has much less complex tasks than our benchmark (e.g. 8-bit adder vs Conway's Game of Life), and (c) being Verilog, is essentially a highly-constrained code-generation task evaluable entirely with unit tests, versus our open-ended hardware generation task that necessitates physical construction and expert evaluation.  Given that the work we report here is at a greater scale, scope, difficulty, and includes an expert-evaluation, and given the above concerns, this paper merits substantially higher scores than a hard reject (2 soundness / 3 excitement).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper provides a framework for design real-world electronic devices based on natural language instruction and goes through the entire process ending up with six different physical devices.\n* The paper generally defines evaluation metrics and experiments well defined and laid out in clear and understandable manner.\n* The dataset contributions could be useful for the community to build upon. \n* The authors ensure that limitations, cataloging of different of errors, and the importance of human expertise is well-represented for different aspects of evaluation and construction of physical devices.", "reasons_to_reject": "* While the paper includes a discussion on code generation with LLMs, it does not touch on code design relevant to electronic circuits specifically. It would be good to get a better understanding of how the paper distinguishes itself from the relevant literature in that aspect.\n* The authors could have provided a more thorough analysis of the time saving provided by using LLMs. This is especially true in cases where iterative refinement of inputs was needed.\n* The utility of the Micro25 dataset is limited if human evaluation is still required to assess performance.\n* Some additional details on how the LLMs were used would be helpful.\n* Since human expertise is required in various parts of the process, reproducibility is more challenging.", "questions_to_authors": "* Question A: You say that the LLMs are instruction fine-tuned. What kind of instructions did you use and how?\n* Question B: Are there any procedures for what happens when the LLM suggests a part that is not available?\n* Question C: You said that the devices were synthesized by a domain expert that had to correct some common errors. You also state that the LLM's capabilities are comparable to a student from an undergraduate course in microelectronics. Does the LLM reach undergraduate capabilities with or without the corrections? I am still to get a better understanding of the true capabilities of the LLMs.\n* Question D: Can you provide the prompts that you used to design the six real world use cases?", "paper_topic_and_main_contributions": "The paper studies the capabilities of commercial LLMs to provide components for the design of electronic devices starting from textual descriptions. To that end, the paper also releases two relevant datasets (PINS100 and Micro25) that include relevant data required microelectronics design that span all required components to design fully functional microelectronic devices. The paper also proposes a set of evaluation metrics for evaluating the performance of LLMs in microelectronic design and showcases six real-world use cases of devices built from LLM outputs. The authors first describe the general problem setting and their contributions in the introduction and subsequently provides a discussion of related work in arduino devices and LLM based code generation.\n\nNext, the authors describe their first set of experiments and analysis where they assess the capability of LLMs to generate accurate pinout information for 100 common electronic components included in their PINS100 dataset. The second experiment focuses on circuit generation, which constitutes a significantly more complex task that involves four different element that jointly form the full circuit. The authors assess different LLMs' capabilities with a PASS@1 metric and provide an analysis of the various LLMs across the different design cases included in Micro25 dataset.\n\nThe authors final set of experiments involve creating real-world devices based on the output of different LLMs following several rounds of refinement of specification prompts, which are then created by a human using common manufacturing processes for circuits. The authors show pictures of the six different devices and describe their general capabilities, as well as a series of challenges related to correcting different errors in the process through a human expert. Lastly, the authors provide a discussion of different challenges and opportunities (prompt sensitivity, manual evaluation, infusing LLMs with common sense knowledge) along with a conclusion and an extensive discussion of limitations.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The paper could be improved by providing additional discussion and references related to using LLMs for circuit design, such as:\n\n* Thakur, Shailja, et al. \"Benchmarking Large Language Models for Automated Verilog RTL Code Generation.\" 2023 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 2023."}}, {"review_id": "D8hxX8ThMx", "reviewer": "Reviewer_bhLT", "summary": "", "questions": "First and foremost, I commend the authors for their innovative approach to utilizing LLMs. The paper is well structured and the contributions are interesting. However, in order to fully understand the scope, implications and future direction of this research, I have several questions:\n\nQuestion A: Benchmarks: Can you elaborate on the metrics used to design benchmarks PINS 100 and MICRO 25? How are these benchmarks validated to ensure they adequately measure the capabilities they are designed to test?\n\nQuestion B: Error analysis: The paper mentions error analysis for open generation experiments in an appendix. Can you highlight the most common types of errors encountered and speculate on their underlying causes?\n\nQuestion C: Human intervention design: Given that the physical creation of equipment is manual, how do you foresee the integration of automated manufacturing processes in this pipeline? Is it feasible to have language models to guide such processes in the future?\n\nLooking forward to your answers, they will help me understand the research more fully.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and thoughtful review. \n\n**Summary of rebuttal:**\n\n1. **Review text is a 5, review score is a 2 (Review score does not match text).** \\\nTo quote this review: \n- \"pushes the boundaries of what NLP can achieve, showing the potential in the field of NLP extends far beyond routine NLP tasks\" \n- \"the introduced benchmarks will be an invaluable resource, enabling researchers to evaluate and improve the state-of-the-art in this emerging field\" \n- \"this papers approach has the potential to democratize electronic device design, making it available to those without a deep expertise but with a clear idea of vision\" \n\n**We showed this review to 4 other colleagues not involved with this work, who all said the score based on the text should be a 5 (strong accept), not a 2 (hard reject).  We are frankly stunned at the score.**  \n\n*Additional quotes:*\n\n\"thorough performance analysis\", \"impressive results\", \"detailed case studies\", \"practical demonstrations of the model's capabilties [and] also offer insights into the real-world applications of such a technology\", \"[also delve deep into a qualitative assessment of model performance\", \"novel application of NLP\", \"will undoubtedly serve as the basis for future research\", \"six case studies not only validate the thesis' claims, but also provide practical insights into the practical application of the proposed methodology\", \"opens a clear path for future research\"\n\n\n**Rebuttal to Reasons to Reject:**\n\n1. **\"Design of textual descriptions: The generation of high-level descriptions is done by experts, and the LLMs essentially knowledge is code generated, which makes me question the potential of end-to-end automation.\"** \\\n*Response:* This is not true -- the high-level descriptions of the devices are in natural language, not expert generated (e.g. for the emoji keyboard, it's `create a keyboard that plugs in as a USB device, but instead of a full keyboard it has only a small number of buttons. The keyboard should only have buttons for 9 popular emojis.`).  We will clarify this in the description. \n\n2. **Simple device range: The complexity of the devices selected for this article is limited, most devices have less than 50 lines of code and are controlled by a microcontroller with 2K memory. This gives me some concerns about practical applicability and scalability.** \\\n*Response:* This is a criticism of the entire field of microcontroller-based electronics, not of this paper.  **Nearly every single non-trivial electronic device created after the 1990s contains one or more microcontrollers, and nearly all of those microcontrollers contain less than 2k of RAM.**  Embedded applications are bare-metal (i.e. without an operating system) and do not generally have large RAM requirements -- indeed, higher RAM only became available in microcontrollers when they started to be used for digital signal processing (e.g. audio) applications.  A keyboard, a mouse, indeed a visible light spectrometer all have modest RAM requirements and easily fit.  **Indeed, the ATMega328p targeted in this work is widely believed to be the most popular microcontroller in the world.** \\\n\\\nSimilarly, \"most devices [requiring generating] less than 50 lines of code\" is not a valid criticism -- the entire field of code generation could not fit 50 lines of code into the entire context length of its best models only 6 months ago!  HumanEval (Chen et al., 2021), the most popular code-generation benchmark, generally requires generating less than 5 lines of Python code per problem.  *Generating 50 lines of error-free microcontroller code in C in response to a short natural language task description is a truly incredible achievement, like from science fiction!* \n\n3. **Human intervention: Almost all devices contain errors that require the intervention and correction of domain experts. However, the uncertainty of human intervention to reproduce the entire job can become very large.**\\\n*Response:* This is also a criticism of the entire field of code generation, not this paper.  **Every known system (e.g. the extremely popular Github Copilot coding assistant) is deployed as an assistant that requires a human to check and correct the work.**  Further, our system shows extremely high performance without assistance (Experiments 1 and 2), and uses a domain expert to quantify performance on very challenging devices (Experiment 3), much as a Github Copilot would be used in real-world scenarios.\n\n4. **Lack of comparative analysis: While this paper presents benchmarks and presents results, there is a lack of comparison with other potential approaches and no in-depth exploration of why the use of LLMs is superior or necessary for this task.**\\\n*Response:* **Re: comparison**, to the best of our knowledge, this is the very first paper do perform this task, so it is not possible to compare against previous work.  **Re: motivation**, we provide the following text in the paper (L056-L061): *\"Our focus in this work is on automating the task of transforming a high-level concept into a practical electronics schematic, complete with companion microcontroller code—a task that currently requires significant human expertise and effort.\".*  We can expand this to highlight the massive time savings (e.g. our undergraduates typically take several days to design and complete a Simon Says memory game (from MICRO25) in an Arduino -- the system reported in this paper can do this in under a minute). \n\n\n**Clarification Response to questions for authors:**\n\n**Question A: Benchmarks: Can you elaborate on the metrics used to design benchmarks PINS 100 and MICRO 25? How are these benchmarks validated to ensure they adequately measure the capabilities they are designed to test?**\\\n*Response:* This is a good question.  The PINS100 benchmark components are validated by being all high-frequency (i.e. commonly occurring) core components found from electronics simulators and popular tutorial websites (e.g L197-L198). The MICRO25 benchmark are validated by all being core skills taught across many undergraduate microcontroller curricula (including a popular undergraduate course taught for a decade by the authors).  \n\n**Question B: Error analysis: The paper mentions error analysis for open generation experiments in an appendix. Can you highlight the most common types of errors encountered and speculate on their underlying causes?**\\\n*Response:* The error analysis in the Appendix already includes these, but we could highlight these in the main body of the paper using the extra page, if that's what you're asking?  For example, GPT4 favors using common parts and libraries, while Claude-V1 likes to design things manually from first principles (which is much more difficult), which may contribute to the difference in performance. \n\n**Question C: Human intervention design: Given that the physical creation of equipment is manual, how do you foresee the integration of automated manufacturing processes in this pipeline? Is it feasible to have language models to guide such processes in the future?**\\\n*Response:* This is a good question -- and we do indeed foresee this (which is why the excitement of this work is so high).  It is easily foreseeable that using a model that constraints its generation to the components available in a large pick-and-place machine, that is combined with a circuit board mill, printer, or other device, that one could go from natural language description to physically-manufactured-device in less than an hour within a few years time. \n\n**Missing references:**\n\nThank you for pointing us to Chip-Chat (Blocklove et al., Arxiv 22 May 2023), which is considered a contemporaneous work by ACL policy ( https://www.aclweb.org/adminwiki/index.php/ACL_Policies_for_Submission,_Review_and_Citation ). We will include it in our related work.  That being said, comparing our work to the work you have suggested: \n\n1. *25 vs 8 tasks, task complexity:* We introduce two benchmarks (one with 100 components, another with 25 designs), whereas Chip-Chat's benchmark includes only 8 designs (pg. 3 Table 1), many of them far simpler than our benchmarks (e.g. an 8-bit shift register; though he dice roller is nearly comparable to our random number generator without the radiation sensor for the random seed, or the nixie tube driver). \n\n2. *Verilog vs Arbitrary Hardware:* Chip-Chat is working in Verilog for FPGAs, and being Verilog, is essentially a highly-constrained code-generation task evaluable entirely with unit tests, versus our open-ended hardware generation task that necessitates physical construction and expert evaluation. \n\n3. *6 vs 1 case study:* Chip-Chat paper includes a single case study where the system is used in the role of an assistant (the 8-bit accumulator-based processor), our paper includes *6*. \n\n4. *Paper length requirements*: The IEEE (?) format allows the Chip-Chat paper to have what we measured to be 11k words (50k characters). The ACL format gives us ~ 5500 words (~30k characters), approximately half that of the Chip-Chat paper.  While our paper introduces benchmarks that are between 3 to 6 times greater in scope than ChipChat, the ACL paper length requirements necessitate that some details (like the error analysis) must be in the Appendix. \n\n**Remarks on Reproducibility:** \n\nThis is a work of open source science, and every effort has been made to make the resulting research products easy to use and well-documented for reproducibility.  The code, benchmarks, libraries, and results currently sit in a Github repository awaiting being made public.  Runscripts allow users to regenerate the results in about 12 hours (at current API speeds).  Unlike other tasks that require evaluation by domain experts (e.g. medical tasks, where doctors might disagree that a given symptom is individative of one diagnosis versus another), in our tasks the evaluations are objective -- i.e. the LED either blinks, or it doesn't; the servo motor either oscillates back and forth between 45 and 135 degrees, or it doesn't.  Papers that require (a) domain expertise, or (b) API access are still considered highly replicable (the PIs advisors used to have to travel across the world 20 years ago to access specific Linguistic Data Consortium (LDC) corpora to replicate others work).  Where possible and applicable, the results of this work are augmented with additional information (e.g. screenshots of the simulator, that augment the JSON netlists produced by the model) for usability and to quickly aid in verification.  \n\nThe reproducibility score suggests this work is not reproducible, but we believe this to be in error as every effort has been made to make this work  reflective of *highly reproducible* open source science. \n\n\n**Updated Scores:**\nBased on the truly extraordinary praise in the original review, along with the detailed comments addressed above, we request the reviewer update their scores from hard reject (2/3) to accept.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Innovative Applications: This article shows a novel application of NLP, which is more innovative in transitioning from pure text processing to tangible electronic device design.\n\nEmpirical Results: Empirical results, especially the GPT-4 model, performed well on benchmarks such as PINS 100 and MICRO 25.\n\nIntroduction of new benchmarks: The introduction of benchmarks such as PINS 100 and MICRO 25, which provide standardized evaluation metrics for evaluating models in this novel application, will undoubtedly serve as the basis for future research.\n\nBeneficial case studies: The inclusion of six case studies not only validates the thesis's claims, but also provides practical insights into the practical application of the proposed methodology.\n\nComprehensive analysis: This paper not only highlights the advantages and potential of language models in electronic design, but also acknowledges their limitations, opening a clear path for future research.\n\nBenefits to the NLP community:\n\nExpanding horizons: This paper pushes the boundaries of what NLP can achieve, showing that the potential in the field of NLP extends far beyond routine NLP tasks.\n\nImplications for interdisciplinary research: By combining NLP with electronic design, this work can inspire further interdisciplinary research.\n\nResource Contribution: The introduced benchmarks will be an invaluable resource, enabling researchers to evaluate and improve the state-of-the-art in this emerging field.\n\nDemocratizing Design: If further refined, the paper's approach has the potential to democratize electronic device design, making it available to those without deep electronics expertise but with a clear idea or vision.\n\nIn conclusion, this paper is a good way to encourage people to think outside the traditional scope of NLP and consider how to use NLP to empower various fields.", "reasons_to_reject": "While this article describes the new application of LLMs in electronic devices, there are several issues that make me question:\n\n1. Design of textual descriptions: The generation of high-level descriptions is done by experts, and the LLMs essentially knowledge is code generated, which makes me question the potential of end-to-end automation.\n\n2. Simple device range: The complexity of the devices selected for this article is limited, most devices have less than 50 lines of code and are controlled by a microcontroller with 2K memory. This gives me some concerns about practical applicability and scalability.\n\n3. Human intervention: Almost all devices contain errors that require the intervention and correction of domain experts. However, the uncertainty of human intervention to reproduce the entire job can become very large.\n\n4. Lack of comparative analysis: While this paper presents benchmarks and presents results, there is a lack of comparison with other potential approaches and no in-depth exploration of why the use of LLMs is superior or necessary for this task.\n\nIn conclusion, while this paper touches on the innovative intersection of NLP and electronics, more comprehensive research, broader scope, and rigorous validation are needed.", "questions_to_authors": "First and foremost, I commend the authors for their innovative approach to utilizing LLMs. The paper is well structured and the contributions are interesting. However, in order to fully understand the scope, implications and future direction of this research, I have several questions:\n\nQuestion A: Benchmarks: Can you elaborate on the metrics used to design benchmarks PINS 100 and MICRO 25? How are these benchmarks validated to ensure they adequately measure the capabilities they are designed to test?\n\nQuestion B: Error analysis: The paper mentions error analysis for open generation experiments in an appendix. Can you highlight the most common types of errors encountered and speculate on their underlying causes?\n\nQuestion C: Human intervention design: Given that the physical creation of equipment is manual, how do you foresee the integration of automated manufacturing processes in this pipeline? Is it feasible to have language models to guide such processes in the future?\n\nLooking forward to your answers, they will help me understand the research more fully.", "paper_topic_and_main_contributions": "This paper delves into the groundbreaking exploration of LLM capabilities, particularly in the field of electronic devices design. Traditionally, LLMs have been praised for their proficiency in understanding and generating textual content. However, this research reveals a new aspect of these models: the ability to design functional electronic circuits based on high-level textual descriptions. This capability has also been confirmed by some recent work, such as Chip-Chat (https://arxiv.org/abs/2305.13243).\n\nMain Contributions:\n\n1. Benchmarks Introduction: The authors introduced two distinct benchmarks to assess the abilities of language models in electronic design: PINS 100 This benchmark evaluates the model's understanding of various electrical components. MICRO 25: A more comprehensive benchmark that assesses a model's capability to design common microcontroller circuits, accompanied by relevant code within the Arduino ecosystem.\n\n2. Performance Analysis: The paper presents a thorough performance analysis, noting that models such as GPT-4 and Claude-V1 achieved impressive results, ranging from 60% to 96% on the task of generating complete devices.\n\n3. Case Studies: Six detailed case studies are provided, showcasing the potential of language models as design assistants for moderately intricate devices. These case studies not only serve as practical demonstrations of the model's capabilities but also offer insights into the real-world applications of such a technology.\n\n4. Qualitative Analysis: The authors delve deep into a qualitative assessment of model performance, highlighting the challenges in evaluation and offering recommendations for future advancements in complex circuit design.\n\nEssentially, this paper stands at the crossroads of natural language processing and electronic design, presenting a new research perspective. Thanks to the authors for showing us the untapped potential of some LLMs.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None.", "Typos_Grammar_Style_and_Presentation_Improvements": "I think it would be a lot better to add some contrast to other jobs.", "Justification_for_Ethical_Concerns": "None."}, "extra_text": {}}, {"review_id": "c0PkxWxVN5", "reviewer": "Reviewer_8oFb", "summary": "", "questions": "A. Could you provide an example over which exact parts are considered critical and non-critical on strict vs. permissive evaluation.\n\nB. Is ground truth or references also provided in the benchmarks?\n\nC. The high performance of the models seems to indicate that the pretrained language models have been exposed to such problems during training. This is also indicated by the authors in lines 337-345. Did the authors investigate on what extend the pretrained language models could have been exposed to inputs and outputs such as those included in the benchmarks? \n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We respectfully strongly disagree with this review:\n\n- **Code (and electrical schematics) cannot be evaluated with n-gram overlap:**\\\nCode **can not** be evaluated using n-gram overlap with reference code, because there are many possible solutions that produce the same output -- this is why code is evaluated by running it, and testing its output.  The exact same situation applies to electrical circuits -- there are many possible schematics that can produce a given output.  There are no existing methods to perform unit tests on hardware generation tasks such as this, so evaluation must be completed manually by a domain expert.\n\n- **GPT-4 performing well at a benchmark does not prohibit publication**:\\\nThis does not follow precident.  GPT-4 meets or exceeds human performance on many popular benchmarks -- for example, GPT-4 answers (and explains) nearly all 4th and 8th grade standardized science exam questions correctly, yet we still use these standardized benchmarks to measure model performance -- and so too should we on electronics tasks, where strong performance would reflect being a highly successful student (95%+ percentile) in our undergraduate microcontroller class.  Similarly, while it's accepted that GPT-4 outperforms nearly all other models on nearly every task, as we see in this paper, even the next-most-performant model (Claude-V1) achieves only 60% on schematics and 76% on code, while open models are surely much less capable.  Indeed, GPT4 performance is often seen as a ceiling to be pursued for open source models (such as the plethora of code generation models released only in the past 2 weeks seeking to meet GPT4 performance on HumanEval), not a sign to discontinue research in a subfield. \n\n- **Evaluation details are provided throughout; experiments completely reproducible**:\\\n\"It is unclear how the evaluation was conducted, and whether it is possible to be reproduced consistently by future work\".  Details of the experiments and evaluation are provided throughout the paper and Appendix.  As described in the paper (Introduction, Appendix), all data, code, results, and runscripts are provided in a Github repository that will be made public upon acceptance, and the data can be regenerated in about 12 hours (plus whatever time the domain expert requires for their evaluation).  While this domain requires expert evaluation, that evaluation is highly objective to a domain expert: a circuit either successfully blinks and LED, or it doesn't; a servo motor either successfully oscillates between 135 and 45 degrees, or it doesn't.  The results are eminently reproducible, and if in doubt, you are invited to submit the prompt from Table 6 to GPT-4 8k, substituting in the following platforms and  task descriptions from MICRO25 (TSV below), to voice any valid criticisms of the objectivity of the evaluation empirically:\n\n```\nmicro25.tsv\ngeneralArea\tspecificArea\tplatform\ttaskName\ttaskDescription\nInput\tDigital - Button\tArduino Uno\tinput-digital-button\tCreate a device with a single push button, that turns on an I/O pin when the button is pressed, and turns off that same I/O pin when the button is not pressed.\nInput\tDigital - Multiple Buttons\tArduino Uno\tinput-digital-multiple-buttons\tCreate a device with 4 push buttons, that turns on an I/O pin when exactly 2 of the buttons button are pressed, and turns off that same I/O pin otherwise.\nInput\tAnalog - Potentiometer\tArduino Uno\tinput-analog-potentiometer\tCreate a device that reads the analog input from a potentiometer configured as a voltage divider and sourced by 5V.  When the input is greater than or equal to 2.5V, it should turn on an I/O pin, while if the input is below 2.5V it should turn off the I/O pin.\nProtocols\tSerial\tArduino Uno\tprotocols-serial\tCreate a device that reads the Serial port at 9600 baud.  Whenever the string \"hello\" is transmitted to the device, it will respond by sending \"world\".\nProtocols\tI2C\tArduino Uno\tprotocols-i2c\tCreate a device that reads a value from an I2C device, and displays that value to the serial port every second.  The device address is 0x50, the register to read is 0x15, and the value is 8 bits long.  The value should be displayed in base 10.\nProtocols\tSPI\tArduino Uno\tprotocols-spi\tCreate a device that connects to another device using SPI.  Whenever the string \"hello\" is transmitted to the device, it will respond by sending \"world\".\nProtocols\tSD Card\tArduino Uno\tprotocols-sd\tCreate a device that opens a file called \"out.txt\" on an SD card, and and every 10 seconds, randomly prints one of the following animal names (as well as a newline character): cat, dog, mouse, parrot.\nOutput\tMotor - DC\tArduino Uno\toutput-motor-dc\tCreate a device that oscillates between spinning a DC motor one direction then the other direction every 5 seconds. \nOutput\tMotor - Stepper\tArduino Uno\toutput-motor-stepper\tCreate a device that oscillates the output of a stepper motor clockwise then counterclockwise 45 degrees, every 5 seconds.\nOutput\tMotor - Servo\tArduino Uno\toutput-motor-servo\tCreate a device that contiuously moves a hobby servo back and forth from 45 degrees to 135 degrees every 5 seconds.\nOutput\tLED - Blink\tArduino Uno\toutput-led-blink\tCreate a device that blinks an LED every 500 milliseconds. \nOutput\tLED - Sequence\tArduino Uno\toutput-led-sequence\tCreate a device with 4 LEDs, that blink in sequence, one after the other, every 500 milliseconds.  When the end of the sequence is reached, the pattern should reset, and continue indefinitely. \nOutput\tLED - Neopixel\tArduino Uno\toutput-led-neopixel\tCreate a device with a neopixel (WS2812) that slowly and continuously cycles a rainbow of colors.\nOutput\tRelay\tArduino Uno\toutput-relay\tCreate a device that turns a relay on for 2 seconds, then off for 5 seconds, continuously.  The relay coil takes 500 milliamps of current to engage.\nOutput\tLCD - Raw\tArduino Uno\toutput-lcd-raw\tCreate a device that prints the phrase \"Hello World\" on a 16x2 LCD.  Use a HD44780 compatible 16x2 LCD, configured normally (i.e. without an I2C, Serial, or other simpler connection). \nOutput\tSound - Buzzer\tArduino Uno\toutput-sound-buzzer\tCreate a device that continuously plays a high, medium, then low tone on a Piezo Buzzer, changing tones every second.\nOutput\t7 Segment Display\tArduino Uno\toutput-7segment\tCreate a device that counts from 0 to 9 on a 7 segment display, indexing numbers every 500 milliseconds.  When the end of the cycle is reached, it should start again.\nOutput\tAnalog Output\tArduino Uno\toutput-analog\tCreate a device that produces a sawtooth wave, ramping up from 0V to 5V every 50 milliseconds. \nSensors\tResistive - CDS Light Resistor\tArduino Uno\tsensors-resistive-cds-cell\tCreate a device that reads the value from a CDS cell, and outputs it to the Serial port every second. \nSensors\tManual Protocol - Ultrasonic\tArduino Uno\tsensors-ultrasonic\tCreate a device that reads the distance from a HC-SR04 ultrasonic distance sensor, and outputs the distance (in centimeters) to the Serial port every second.\nSensors\tI2C Protocol - Magnetic\tArduino Uno\tsensors-magnetic\tCreate a device that reads the current magnetic field readings using a HMC5883L magnetometer.  The readings (x, y, z, and total field strength) should be output to the Serial port every second.\nLogic\tInteractive - Simon Game\tArduino Uno\tlogic-simon\tCreate a device that implements the popular memory game simon, where users enter progressively longer sequences of colors.  It should have 4 possible colors, and include sound when the button is touched, as well as when winning/losing.  The game should timeout if the user doesn't enter input after 5 seconds.\nLogic\tNon-interactive - Conway's Game of Life\tArduino Uno\tlogic-conway\tCreate a device that implements the popular Conway's Game of Life, on an 8x8 LED matrix.  The game steps should cycle every 500 milliseconds.  If the board is empty, it should randomly initialize the game again.  There should be a pushbutton that allows randomly resetting the game. \nLogic\tInteractive - Clock\tArduino Uno\tlogic-clock\tCreate a device that implements a clock that prints the current time on a 16x2 character LCD display with an I2C interface.  It should have three buttons to help set the time in a user frieldly way: one to increment hours, one to increment minutes, and one to increment seconds.  The timekeeping should be performed by the Arduino, and not an external real-time clock.\nLogic\tInteractive - Air Temperature\tArduino Uno\tlogic-air-temperature-meter\tCreate a device that reads the current air temperature, and displays it as a color on a neopixel (WS2812). The color should be fully blue at 0C, fully red at 30C, and an interpolation of blue and red between those temperatures.\n``` \n\nWhile the paper mentions (in no less than 13 separate locations, by our count) details of the domain expert evaluation, we will include the additional requested details of the domain expert in the Appendix: \"The electrical engineering expert used for evaluating this work is a co-author of this paper, award-winning science educator, prolific open source hardware author, and internationally-recognized researcher with approximately 50 articles describing their open source hardware work in popular international news media such as Reuters, Forbes, and the Washington Post.  The benchmarks described in this work were both authored and evaluated by the domain expert as reflective of the content of a popular full-term (4 month) undergraduate course in microcontroller design for computer science students, typically taken in a student's final year of undergraduate studies at a R1 (\"Very high research activity\") institution in the United States.  The domain expert has delivered the course approximately 10 times over a decade, to approximately 500 students.  Students in this computer science course have competed in a design competition against engineering students -- and won each year -- validating the scope and quality of this curriculum.\"\n\n**Questions For The Authors:**\n\n- **Question A: Example of non-critical pin:**\\\n*Response:* An example of a non-critical pin would be the DRDY (data-ready) pin on an HMC5883L magnetometer.  The magnetometer is an I2C device that nominally requires only 4 wires to function (if wired on a breadboard): SDA, SCL, VDD, and GND.  The DRDY pin is marked as optional (not required) on the datasheet, as connecting the DRDY pin only allows faster polling rates by signifying when measurements are ready to be read by the host microcontroller.  We will include such an example in the Appendix.\n\n- **Question B: Are ground-truth/references provided in the benchmarks:**\\\n*Response:* For code generation/schematic generation tasks, there are no ground truths -- many solutions are possible, which is why their output (rather than their code) is evaluated.  For example, even the most trivial task (blinking an LED) could place the LED on any of 20 I/O pins, use any resistor in the range of 100 ohms to 470 ohms, use a variety of different colors of LED with different forward voltages, write the code in a way that directly through libraries (e.g. digitalWrite()) or indirectly through registers (e.g. PORTB) controls the I/O pin, that delays through blocking (e.g. delay() ) or non-blocking (e.g. watchdog timer/interrupts) methods, etc. -- and this covers only canonical solutions.  A non-canonical solution might control the LED through a transistor, opto-coupler, or relay to allow a wider voltage range, or even a specialized LED driver.  Indeed, we have observed that GPT-4 tends to prefer canonical solutions, while Claude-V1 prefers to create unusual solutions (presumably from first principles).  Given the open-ended, non-expert nature of the user interface (i.e. we want the user to be able to say \"Make me a device that blinks an LED every 5 seconds\", not \"Make me a device that connects an LED with a 2V forward voltage to pin D6 for PWM control, through a 220 ohm current limiting resistor, with code that uses the digitalWrite() interface\"), making highly-constrained references is nether possible nor desirable. \n\n- **Question C: The high performance of GPT-4 suggests similar tasks may be present in the training data.  Did the authors examine this?**\n*Response:* Yes, we exactly examined this, as one of the primary motiviations for Experiment 3.  Quoting the first sentence of Section 5 (\"Experiment 3: Open Device Generation\"): **\"While we’ve observed that language models have the capacity to design comparatively simple devices in Experiment 2, this result is tempered by these benchmark tasks being representative of fairly common skills and capacities that are frequently taught in microcontroller-oriented curricula found in books, internet tutorials, and blog posts – and as such, the MICRO25 tasks likely exist in some form in the voluminous (but closed) training data of these models. Here, we examine how well the best-performing model, GPT-4, can create comparatively more complex devices in a more realistic and qualitative setting [...]. To further increase task difficulty, each of the device specifications in this experiment were explicitly crafted to be highly unusual – either using uncommon components, or combining common components in unusual ways – such that the likelihood of similar devices appearing in the closed model training data is low. The six devices are shown in Figure 4.\"**\n\nWhile the training data to GPT-4 is closed, we can infer that the relative difference in performance between Experiment 2 and Experiment 3 (i.e. 24/25 benchmark tasks generated without error, versus 2/6 case studies generated without error) suggests that training data may play a large part of the performance difference.  Though, as detailed in the **2.5 page detailed error analysis for the case studies provided in the Appendix**, performance is affected by many factors, from intuitive factors (such as that adding many subtasks, such as keys playing music for the Emoji keyboard, affects performance) to unintutive factors (e.g. prompt sensitivity -- i.e. small changes in the prompt causing GPT-4 to fail to include a high-voltage power supply for the Nixie Tube display in the random number generator).  We can similarly infer that GPT-4 has never seen the datasheet or driver for the Hamamatsu C12666MA microspectrometer (Appendix D.3), given that all the timings it generated in the device driver were incorrect, but it clearly has seen linear image sensors before, because they all follow similar protocols for clocking data out, and the driver was mostly correct at a conceptual level if not at a timing level.  \n\n**Overall comments:**\n\nOur overall concerns for this review -- providing a score of *1* for what, objectively, is -- as Reviewer 2 said best -- a paper that \"pushes the boundaries of what NLP can achieve, showing the potential in the field of NLP extends far beyond routine NLP tasks\" and that \"... has the potential to democratize electronic device design, making it available to those without a deep expertise but with a clear idea of vision\", is succinctly described by a colleague not involved with this work but who read this review: \n\n```\nIt is maddening to me that a well-evaluated incremental piece of work, where datasets and methodologies are plentiful but the work is incredibly boring, is rated more highly than imaginative and innovative work blazing a new trail – where by definition evaluation needs to be rethought. It’s a common problem with reviewing, to the extent that ACL has explicitly instructed reviewers not to succumb to this bias ( https://2023.aclweb.org/blog/review-acl23/ ).\n```\n\nIs generating the designs (either directly, or through an assistant interface) for full electronic devices purely from high-level natural language descriptions of those devices an exciting, enabling technology?  We think so. \n\nDoes this work perform sound, well-evaluated, reproducible science?  We would robustly defend \"yes\".  Does this paper perform work at scale that is representative of the field?  As the contemporaneous references provided by Reviewers 1 and 2 show (Thakur et al., 2023; Blocklove et al., 2023), the scale of this work and evaluation is *three times* larger than contemporaneous work.  We would remind potential reviewers that this is the /first/ paper for this subfield, not the last paper, and by definition it takes time for research methods and infrastructure to be rethought to enable the work at great scale.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper presents benchmarks on a novel task, related to text-to-code synthesis.", "reasons_to_reject": "- The high performance of the models suggest that the benchmarks contain little challenge for future work to investigate and expand upon. This could indicate that the task is trivial or that the dataset is not representative of the task's difficulty. The case study inputs are claimed to be more challenging, but those are not properly evaluated.\n- It is unclear how the evaluation was conducted, and whether it is possible to be reproduced consistently by future work. \n- Lines 317-321, and elsewhere, suggest that this was a form of human evaluation by an expert. In that case more information should be included in the paper, e.g. was it a single expert, what is their expertise and their relation to the paper, what steps were taken to eliminate their bias in the study, and if and how was inter-annotator agreement determined.\n- The adoption of the pass@k metric, which was proposed for evaluating the functionality of code over a set of unit tests, seems inappropriate for these benchmarks. In essence, what is measured seems more akin to accuracy or exact match scores. Given the nature of the output, n-gram overlap metrics may be appropriate (provided that the benchmark also provides references).\n- The merits of the case study are very unclear, specifically how physically manufacturing expert-corrected devices can quantify the pretrained language models' capabilities.\n", "questions_to_authors": "A. Could you provide an example over which exact parts are considered critical and non-critical on strict vs. permissive evaluation.\n\nB. Is ground truth or references also provided in the benchmarks?\n\nC. The high performance of the models seems to indicate that the pretrained language models have been exposed to such problems during training. This is also indicated by the authors in lines 337-345. Did the authors investigate on what extend the pretrained language models could have been exposed to inputs and outputs such as those included in the benchmarks? \n", "paper_topic_and_main_contributions": "This paper introduces two benchmarks, related to text-to-code tasks: one for assessing a language model's capability to generate electronic component specifications from a brief textual requirement, and one for assessing the more complex task if fully designing a micro-controller circuit including a list of materials, the component's pinouts, the schematic, and code, from a brief textual requirement. All non-code output is represented in the json format. The datasets consist of 100 and 25 test instances each. \n\nThe paper examines the performance of pretrained language models (GPT-3.5, GPT-4, and CLAUDE-V1) on the proposed benchmarks, and shows they achieve up to 86% and 94% pass@1 scores respectively. The paper also includes six case studies over more challenging inputs, but those are not directly evaluated, but instead corrected by an expert and constructed as physical devices.", "excitement": 2, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- Figure 3 should be closer to page 5 where it is referred to.\n"}, "extra_text": {}}, {"review_id": "PC0j9dsM57", "reviewer": "Reviewer_zUFP", "summary": "", "questions": "- Question A: In experiment 2, are the 5 types of tasks tested only in isolation? Are any of them related in a way that they would be carried out sequentially by humans? Or would humans carry them out in any order (I am referring to both types of tasks and the specific tasks listed in the table)?\n\n- Question B: What is the relation between the device descriptions in Figure 4 and Appendix B4? Are either of these the input/query descriptions?\n\n- Question C: Were there any issues with the models producing the correct formal output syntax, e.g. valid json or json according to the (prompt) instructions? Are these included in the failure cases when calculating accuracy? It would be interesting to separate this from actual wrong or faulty specifications.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and thoughtful review. \n\n**Rebuttal to Reasons to Reject:**\n\n1. **Evaluation follows best-practices, not superficial:**\\\n**Reviewer 4:** \"The evaluation is only superficial with no further insights into interesting failure (or success) cases. Experiment 1 and 2 are evaluated via accuracy. The evaluation of experiment 3 does not contain any quantification of the problematic cases that are mentioned. Appendix D contains more details but also no numbers.\"\\\n**Response:** The evaluation (using Pass@1, not accuracy) is the standard evaluation based on contemporary research methods for code generation, and the main metric used for most benchmarks including HumanEval from Codex (Chen et al., 2021).  Further, examining detailed coding errors is not standard practice -- even the massive 54 page StarCoder paper ( https://arxiv.org/pdf/2305.06161.pdf ), unconstrained by page limits, includes only a single short paragraph (Page 29, Section 8.3, **154 words**) on the utility of using their model as an assistant, noting that it is \"somewhat capable yet brittle\", and that \"[it] has clear limitations: it sometimes proposes wrong solutions, presents wrong facts, and can make offensive comments\".  **Even still, in contrast to other contemporary work, this work includes a substantially larger error analysis than even StarCoder at 2.5 pages (~ 1500 words) in the Appendix**.  While it is true that the error analysis doesn't include a table of quantitative categories, this is not standard practice for case studies because the total number of samples in each bin would be too low to be helpful (i.e. statistically significant using inferrential statistics).  Instead, case studies -- especially for papers proposing new tasks that require expansive domain-expert evaluation (like medical tasks) -- typically provide a detailed exposition of case studies to motivate their position and analysis, and promote future research in evaluation methods.\n\n2. **Clarity on differences with respect to code generation tasks:**\\\n**Reviewer 4:** \"It is difficult for me to understand the difference of this work to code generation, mainly because I cannot understand the relation of the different outputs to one another (see also Question section). Figure 2 shows 4 different types of output and this diagram suggests they would be generated in order but it is not clear whether they depend on one another and if yes, how the evaluation shows this.\"\\\n**Response:** The main difference between code generation and this device-generation task is that this task requires both code generation and the new task of electrical schematic generation (and, these must be done jointly, so that the code can use the schematic -- e.g. the code trying to turn on an LED on Arduino D3 while the LED is on D4 would not be helpful).  The bill-of-materials and component-pinouts are provided to mirror how humans design circuits (i.e. first you need to know the parts you'll use (bill of materials), and what pins they have (component pinouts) before you can connect them electrically (schematic) and write code that accomplishes the task (code)).  **We welcome (and strongly encourage!) thoughts with respect to clarifying the presentation of the work for a joint NLP + Electrical Engineering audience, and have indeed workshopped a preprint of this paper through a moderate audience to improve its clarity to a broad audience, but do not believe that fixable concerns with clarity on an interdisciplinary paper warrant a hard-reject (2 on soundness).** Similarly, though our task and use case are brand new (and, we believe, truly transformative for de-skilling electronic device design to people with no background in electrical engineering), the ACL reviewing guidelines ( https://2023.aclweb.org/blog/review-acl23/ ) explicitly state that new and interdisciplinary research should not be discounted because of its novelty -- indeed this is what we should be branching towards: \"Heuristic: This has no precedent in existing literature.  Why this is problematic: Believe it or not: papers that are more novel tend to be harder to publish. Reviewers may be unnecessarily conservative.\"\n\n\n**Clarification Response to questions for authors:**\n\n**Question A: In experiment 2, are the 5 types of tasks tested only in isolation? Are any of them related in a way that they would be carried out sequentially by humans? Or would humans carry them out in any order (I am referring to both types of tasks and the specific tasks listed in the table)?**\\\n**Response:** The prompt for Experiment 2 (nominally, what's shown in Figure 3) produces 4 kinds of output: the (a) bill of materials (BOM), (b) component pinouts, (c) electrical schematic, and (d) code.  Of these, (a), (b), and (c) are directly related because (1) every part on the BOM has to be on the schematic, and (2) every component pinout has to be on the schematic.  In this way, we include BOM and pinout generation as a way of promoting \"chain of thought\" prompting (Wei et al., 2022), explicitly asking the model to perform the task in the same pipeline that humans perform it so that much of the information it needs to generate the schematic is already present in the generation.  We describe this in Section 4 (\"Representations\" subsection, L275-L289), but would welcome thoughts on improving this text for a broader audience. \\\nSimilarly, one need only evaluate the schematic to fully evaluate the BOM and the component pinouts (which is why we evaluate only the schematic and code directly).  But, this would not be intuitive to those without an electronics background, and hasn't been caught by  our previous proofreading -- thank you for pointing this out, we will clarify it in the text. \n\n**Question B: What is the relation between the device descriptions in Figure 4 and Appendix B4? Are either of these the input/query descriptions?**\\\n**Response:** **Part 1:** The 6 device descriptions in Figure 4 are the same as those 6 described in more detail in the Appendix B4 (i.e. \"Random Number Generator\" in Figure 4 is the same as B4, \"Emoji USB keyboard\" in Figure 4 is the same as \"Emoji Keyboard\" in B4, etc).  **Part 2:** While the device descriptions have much of the same content as the actual task prompts to generate the devices in the model, they are edited for length.  All task descriptions used to generate all devices in the paper are included in the Github repository.  For example, the task string for the USB keyboard task was: \n\n```\n[keyboard-emoji6a] create a keyboard that plugs in as a USB device, but instead of a full keyboard it has only a small number of buttons. The keyboard should only have buttons for 9 popular emojis, expressed as ASCII characters, not unicode.  One emoji should be a heart.  There should be an LED that's on all the time, but blinks off for 500 milliseconds when a button is pressed.  There should also be a piezo buzzer, that plays a brief tune that is of the same affect as the emoji being pressed -- for example, a love song for the heart emoji, a happy song for a happy emoji, sad music for a sad emoji, and so forth.\n```\n\n**Question C: Were there any issues with the models producing the correct formal output syntax, e.g. valid json or json according to the (prompt) instructions? Are these included in the failure cases when calculating accuracy? It would be interesting to separate this from actual wrong or faulty specifications.**\\\n**Response:** This is a good question.  There were zero errors as a result of generating output in an incorrect format.  This is both because the models are very good to adhering to JSON format prompts, and the few formatting issues they had in pilot experiments (like adding comments that said `\"# ... And so on\"` in generating schematics) were entirely resolved by including a few example generations in the prompt.  While this does make the prompt large (~ 2k tokens, see Table 6 in the Appendix), the benefit is that it's free of generating any syntax errors. We will add this clarification re: zero syntax errors in the text.\n\n\n\n**Typos Grammar Style And Presentation Improvements:**\n\nThank you for sending these comments on presentation.  One of the challenges with writing this paper is that the work is extremely exciting, but it's also for a new and highly interdisciplinary task that blends a field that one normally doesn't see combined with NLP (electrical engineering).  We have already workshopped the paper through a large number of pure-NLP colleagues, and will use your comments to further refine our descriptions.\n\n\n**Remarks on reproducibility:**\n\nThis is a work of open source science, and every effort has been made to make the resulting research products easy to use and well-documented for reproducibility.  The code, benchmarks, libraries, and results currently sit in a Github repository awaiting being made public.  Runscripts allow users to regenerate the results in about 12 hours (at current API speeds).  Unlike other tasks that require evaluation by domain experts (e.g. medical tasks, where doctors might disagree that a given symptom is individative of one diagnosis versus another), in our tasks the evaluations are objective -- i.e. the LED either blinks, or it doesn't; the servo motor either oscillates back and forth between 45 and 135 degrees, or it doesn't.  Papers that require (a) domain expertise, or (b) API access are still considered highly replicable (the PIs advisors used to have to travel across the world 20 years ago to access specific Linguistic Data Consortium (LDC) corpora to replicate others work).  Where possible and applicable, the results of this work are augmented with additional information (e.g. screenshots of the simulator, that augment the JSON netlists produced by the model) for usability and to quickly aid in verification.  \n\nThe reproducibility score suggests this work is not reproducible, but we believe this to be in error as every effort has been made to make this work  reflective of *highly reproducible* open source science.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The work evaluates LLMs in a domain that many NLP experts are unfamiliar with but that is an obvious use case for LLMs that is similar to code generation. Particular domain knowledge is needed to carry out this evaluation, making this paper a great interdisciplinary contribution.\n- The paper is well structured and written and the three experiment setups are well chosen to investigate the question.", "reasons_to_reject": "- The evaluation is only superficial with no further insights into interesting failure (or success) cases. Experiment 1 and 2 are evaluated via accuracy. The evaluation of experiment 3 does not contain any quantification of the problematic cases that are mentioned. Appendix D contains more details but also no numbers.\n\n- It is difficult for me to understand the difference of this work to code generation, mainly because I cannot understand the relation of the different outputs to one another (see also Question section). Figure 2 shows 4 different types of output and this diagram suggests they would be generated in order but it is not clear whether they depend on one another and if yes, how the evaluation shows this.", "questions_to_authors": "- Question A: In experiment 2, are the 5 types of tasks tested only in isolation? Are any of them related in a way that they would be carried out sequentially by humans? Or would humans carry them out in any order (I am referring to both types of tasks and the specific tasks listed in the table)?\n\n- Question B: What is the relation between the device descriptions in Figure 4 and Appendix B4? Are either of these the input/query descriptions?\n\n- Question C: Were there any issues with the models producing the correct formal output syntax, e.g. valid json or json according to the (prompt) instructions? Are these included in the failure cases when calculating accuracy? It would be interesting to separate this from actual wrong or faulty specifications.", "paper_topic_and_main_contributions": "This paper empirically evaluates LLMs' ability to generate electronic designs based on high-level and freeform language descriptions. Three experiments are described that test the model's capability in 3 tasks with increasing difficulty.\n\nThe main contributions are\n- The evaluation task in a domain that is not accessible to the typical NLP researcher. The task targets a series of steps are needed to (later) generate a final product.\n- The application and evaluation of the task using the GPT-4 and Claude models.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- About presentation, I find the many images of devices a bit misleading because no physical objects are being generated. While the images are illustrative, they add little without further explanation. It is impossible for me, who has no background knowledge in electronics, to understand how the formal specifications relate to the images. I only see the relation between free-form input language and image, e.g. for the emoji keyboard, but would need more explanation for the relation between specs and image. In Figure 1, I have no idea at all what I am looking at (I googled nixie tube and then was able to recognize it). Since the focus is on generating specifications, as a reader I would rather like to understand the specs better, even if textual and formal specs seem more boring than images (some of the images can stay of course, or be smaller, or move to the appendix). For example, it is more meaningful to me to see Figure 2 with the 4 boxes illustrating the generation task than Figure 1 that hints at generating an actual circuit. I would recommend merging the two figures.\n\n- Instead of images (or instead of large images since they are taking quite a bit of space) I think one of the concrete examples from the appendix would be great in the main body of the paper e.g. a truncated version of Table 6.\n\n- When referring to appendix sections, please refer to the specific section. If you are using the LaTeX template, this can be done by using labels just like section labels. It helps to find information faster.\n\n- p. 4 \"Results\": \"is shown Table 1\" --> \"is shown in Table 1.\"\n\n- GPT-4 is sometimes spelled GPT4 throughout the paper\n\n- Table 2: mentions \"popular\" games that in fact might not be known so much by everyone, maybe you could provide links to descriptions\n\n- Appendix D, title: \"Modifcations\" -> \"Modifications\"\n\n- D.1, intro is mentioning the appendix but this is the appendix.\n\n- In the main body of the paper, I felt that the important domain-specific vocabulary was explained or exemplified where necessary. In the appendix at times, I could not follow the argumentation because I lack domain knowledge. For example in B1 probably because this wasn't detailed in the paper. I just have no idea of the concept of a pin and what it entails, even with this explanation.\n\n- I liked that all the figures and tables are well formatted with easy-to-read captions and other text!"}}]

Author rebuttal:

Meta-review: As the AC for the paper, I commend the authors and reviewers for an extremely engaged discussion! Thank you to the authors for raising discussion points about quality reviewing practices, and to the reviewers for being receptive and open to these points. My metareview space is limited but I will summarize to the best of my ability.

In general, reviewers found this work to sufficiently support its major claims, although additional support was needed for some minor points. Reviewers found the paper to have strong merits (e.g., an innovative direction) but also key weaknesses (e.g., concerns regarding the evaluation) such that it could benefit from another round of revisions.

**Summary of Reviewer Feedback and Discussion:**
- **Reviewer P9mH** liked the framework provided by the authors, felt that the evaluation metrics and experiments were well-defined and thorough, and thought that the data could be a useful contribution to the community. However, they felt that the paper did not adequately distinguish itself from work relevant to code design for electronic circuits, and that an analysis of time savings from using LLMs could benefit the paper. They felt that the utility of the Micro25 dataset may be limited, requested details regarding the use of LLMs, and noted that reproducibility may be challenging since evaluating the work requires human expertise, in addition to asking clarifying questions. The authors responded that they did not discuss code generation applied to electronic circuits because their work is among the first in this subfield, and that their reliance on human evaluation is a strength. They noted that they are committed to releasing code, benchmarks, library, and experimental results, and that analyzing time savings is beyond the scope of their paper although an interesting direction for future work. They also provided detailed clarifying responses to the other questions.  In their response, Reviewer P9mH thanked the reviewers and clarified that it is important to include relevant papers to the extent possible to help situate the work in the broader research context.
- **Reviewer bhLT** thought the work was novel and appreciated the strong empirical results and new benchmarks. They noted that the inclusion of case studies validated the paper's claims and provided insights into practical applications, and they felt that the analysis was comprehensive and the work promised broad benefits to the NLP community. However, they felt that potential for end-to-end automation and scalability may be limited, were concerned by the lack of comparison with other potential approaches, and wished the paper had explored why LLMs were superior or necessary for this task. They also raised some clarifying questions. The authors responded that the design of high-level descriptions does not require domain expertise and the devices used in the paper reflect devices throughout the field, and that all coding assistants require human intervention. They explained that they did not compare to other approaches since their task was novel. Reviewer bhLT thanekd their authors and asked several follow-up questions, which the authors answered.
- **Reviewer 8oFB** liked that the paper presented benchmarks on a novel task, but felt that the high performance benchmarks may suggest that the task itself was trivial or that the data was not representative of the task's true difficulty. They felt that evaluation details were unclear and requested more information about the human expert, and they felt that the pass@k metric was inappropriate and that the merits for the case study were unclear, along with asking some clarifying questions. The authors disagreed that strong performance suggested task triviality, and noted that experimental and evaluation details were provided throughout the paper and appendix. They provided additional details regarding the human expert, and answered the clarifying questions.  Reviewer 8oFb thanked the authors and elaborated on their concerns, and the authors addressed these concerns.
- **Reviewer zUFP** appreciated the evaluation in a domain unfamiliar to many NLP researchers, and felt that the experimental setups were nicely selected. However, they found the evaluation to be superficial and had difficulty distinguishing the work from work towards automatic code generation. They also asked some clarifying questions. The authors responded that they used the standard evaluation metric for related tasks and that they included an error analysis in the appendix. They clarified the differences between their task and code generation tasks, and responded to the other questions. Reviewer zUFP noted that while the evaluation followed standard practices, they do not agree that it followed best practices, and noted that it would be nice to ultimately see what the task and results tell them about the language model, with NLP-focused insights (e.g., how the results relate to NLP tasks or language model capability).
Decision: Accept-Findings