Package: hardsync
HardsyncWhat problem is this solving?Traditionally, when developing embedded applications, the software and firmware are developed separately. Often, the most time-consuming piece of the entire process is deciding on and implementing a communication protocol between the software and embedded device.How hardsync solves thisHardsync attempts to solve this with a different approach. Instead of writing one set of code on your target embedded system (for example, an Arduino) and one set of code on your computer to communicate with the Arduino, you write asinglecontract and then dynamically generate code which you can import from your firmware and software that handles all the communication for you.You no longer have to worry about encodings, termination characters, error handling, device discovery, or any of the other painful, repetitive, and stupid things about communicating with embedded systems. Hardsync handles all this for you, so you can focus on what matters - your application code.Getting StartedInstall hardsyncFirst, you will need to install hardsync. The easiest way is viapippip install hardsyncWrite your first contractWhat is a contract? A contract is a dead-simple python file that states the requests you want to send to your embedded system, and the responses you expect to receive, with their types:class MeasureVoltage:
    class Request:
        channel: int
        integration_time: float
        samples: int
    class Response:
        voltage: floatOnce you have your contract, run the hardsync code generator to create your device-side firmware and computer-side client you will use to interact with your device:python -m hardsync --contract path/to/your/contract.pyBy default, the generated code will be placed in a folder calledgeneratedinside your current directory. You can override this by specifying the--generated-diroption like so:python -m hardsync --contract path/to/your/contract.py --generated-dir path/to/your/directoryOn the device-side, you now need to upload the generated firmware to your device. This is located in afirmwarefolder inside yourgenerateddirectory. How you incorporate and upload this code will depend on your target.ArduinoThe generatedfirmwarefolder contains a full sketch that can simply be uploaded to the device. In the mainfirmware.inofile, you will see something near the top of the file like this:class DeviceClient : public BaseDeviceClient {
public: 
    double measureVoltage() const override {
        // YOUR measureVoltage CODE GOES HERE
    }
}This is where you put the code that actually measures the voltage. For example, if measuring from one of the arduino's analog pins, you might replace//YOUR CODE GOES HEREwith:double DEVICE_VOLTAGE = 3.3;
double MAXIMUM_ANALOG_VALUE = 1024.0;
int analog_value = analogRead(A0);
double voltage = DEVICE_VOLTAGE * analog_value / MAXIMUM_ANALOG_VALUE;
return voltage;Device discoveryOnce you upload the firmware, you can automatically discover your device using hardsync's built in device discovery tools:python3 -m hardsync discoverAnd you can ask hardsync to keep track of this device's identity for future use. If you work with multiple devices, that's fine too. Hardsync will fall back to auto-discovery if no device identity is specified.Interacting with your device from an applicationTo create a program that interacts with the device, import the generated code:from generated.client import Client
client = Client()

response = client.request_voltage(channel=1, integration_time=0.5)
voltage = response.values['voltage']
print(voltage)Now, running this program will have the python code communicate with the arduino, make it run themeasureVoltage()code you specified, and communicate with the arduino to get the value returned frommeasureVoltage(), and print that to the screen..Re-generating codeWhen re-generating code, make sure to runhardsyncfrom the same directory if you want to overwrite previously-generated files. For safety reasons, by default, hardsync will not overwrite your main sketch or your main application. To override this behavior, simply add the--forceflag.DebuggingIf you encounter an unexpected error and it's not immediately obvious how to fix it from the error message you received, submit a bug report on this repository. Make sure to attach the output ofhardsync dumpto the issue to help with reproducing it and identifying the issue.Supported TargetsAll targets are support both on the client- and device- side. That being said, thearduinotarget are intended to be used on the device-side, and thepythontarget is intended to be used on the host side.arduinopythonDesign philosophyPervasive reasonable defaults for batteries-included operationTotal flexibility - users should be able to override virtually everything.Absolute simplicity - users should be able to specificy the minimum amount of information to get their application working, not be beholden to the frameworkTesting, testing, testing. Everything should be rigorously, repeatedly, and completely tested.ArchitectureThis library is based on simple request/response-based communication. Theclient- this can be the device OR your computer, sends arequestto theserver(which can be either your PC or your device), and the server returns aresponse. This request-response communication is referred to as anexchange, and it is theexchangesthat are the most important element of your contract.Feature RequestThis library is under active development. If you have a feature request, or want to change the priority of planned features (see below), submit an issue on this repository.Planned features (in order of priority)Heavy post-decorating contract validation to ensure that it meets all downstream requirementsTests with good and bad contracts, verifying that everything meets specificationsBreak out config into separate config.py module, heavily testSupport for device-initiated request/response pairsSupport for fixed-size arrays in requests and responsesSupport for binary encodingAdd channel.write wrapper around Serial.print statements to reduce program memory, Serial library flexibilityAdd example with how to override baud rate and device serial numberSupport for multiple response fieldsVariable-size arrays in requests and responsesVerify that generated client-side code is valid pythonSet up CI for automatic testing and publishing to PyPiImplement hardsync.ini file, static configurationSupport for multi-hardsync device discovery on same machineMore tests to verify edge cases, especially around type conversion and multiple argumentsMultiple device channelsAdd generated code testing in sandbox environment to verify it worksAdd example of how to add an additional target to the frameworkSupport for overriding device implementation (see below)"time" typesCasting of non-string request types when received by deviceSupport specifying the output directories with a config file in a projectAdd @asciiencoding, @utf8encoding, and @binaryencoding decoratorsAdd header with number of transmitted bytes as option with @validate_transmission wrapperAdd @jsonencoding decorator and implementationAdd @retryable decorator and implementation to Exchange classAdd retries on non-error response.Add optional timeoutsClient-side async supportAdd a hash of the contract used to generate auto-generated filesAdd verilog targetAdd VHDL targetAdd support for default values in contract, make these optional kwargs with defaults[DONE] Add "ping"-based auto-discovery of serial devices (require runtime pass-in)[DONE] Support for user-defined type mappings, clean syntax for this[DONE] Add function to get all the types used and verify they are all defined[DONE] Add tests to verify this functionality[DONE] Add tests for zero-length requests and responses, ensure void declarations are generated[DONE] Fix bugs in generated device arduino code[DONE] Move exchange to top-level of contract[DONE] Fix code generation so that it generates in the current directory, not the module directory.[DONE] Publish package to PyPi[DONE] Add generated client-side code[DONE] Add "ping" default request/response to firmware + client code[DONE] Add Getting started flow for how to actually use it[DONE] Add "Channel" class to allow users to override baud rate, serial number[DONE] Add a hash of thehardsynccodebase in a comment in all generated code for complete traceability[DONE] Add the version number of hardsync used to generate files in a comment[DONE] Replace exceptions with ERROR logging on receiving a ErrorResponse[DONE] Add INFO logging for sent requests / received responses[DONE] Add INFO logging for generated files being written[DONE] Verify as part of testing that generated arduino code compiles[DONE] Add tests to verify that the command-line tool works as expected[efficiency] Change the size of the C++ Arguments array to be specific to the number of arguments actually present.FAQWhy generated code, and not a universal device library?First, embedded systems (the target for this library) have extremely tight memory constraints. Because of this, a
"universal" library that works with
all
microcontrollers, should it be built, would immediately be useless because it's too large for anyone to include.
Second, even on the client side where memory is less constrained, generating code allows for total flexibility on
the user-side should they wish to modify the code without the friction of contributing to an open-source project
(yes, that friction is still very much a barrier to entry).
It also allows for easy integration with
build systems.Why don't you support X platform?We hope to! I've made it as simple as possible to support a new platform, and I plan to provide specific instructions on how to onboard new platforms that are not currently supported. If you need help onboarding a specific platform, submit an issue on this repository. If you work for a company, feel free to reach out directly if you want better support - I'm happy to provide paid support.Why did you open-source this library?Partly because I hate closed-source tools that try to solve the same problem (I'm looking at you LabView). Partly
because I myself heavily use open-source libraries. And partly for the street cred ;)Where does the name come from?hard: this stands for hardware, but it also implies strictness through usingcontracts.syncis for synchronization - firmware and software for communication fundamentally require and manipulate the same underlying information, and should be in sync with each other. The mechanism to synchronize the two is thecontract.How did you come up with the idea?In dealing with embedded systems for the last decade, I became acutely aware of the pain points people using them experience. I always hated LabView, and always wished there was an easier way to co-develop hardware and firmware. It wasn't until after I took a job as a web developer that I was exposed to many of the concepts underlying this framework, and I borrow heavily from web development in it. It also wasn't until I worked at Meta that I discovered the idea ofinterface description languages, such as Thrift, which aim to solve this problem for process-to-process communication. These three factors conspired together to give me the idea for this framework.Why are contracts written in python? Why not YAML or JSON or just plain text?Programming languages are powerful, and python is perhaps the most powerful programming language in existence. Its dynamic programming capabilities make defining contracts in a python file fairly straightforward, and having a contract defined directly in a programming language allows for overriding and specifying functionality at a level that would simply not be possible in any text-based format.It has minimalist syntax similar to YAML. The language does a remarkable job of "staying out of the way" and letting users express their intent.It is the most accessible language on the planet. Far fewer people know YAML than know python.How does hardsync handle communication setup?Hardync, by default, assumes that your device can communicate over a serial interface. All devices running the hardsync firmware will respond to aPingRequest()with aPingResponse(), and so hardsync attepmts to open available serial devices until it finds one that responds with the expected response. Unless you override it, the baud rate is set to 9600. If you know your device's serial number, you can specify it in your contract to avoid hardsync attempting to open other serial ports.Why don't I put my device serial number in the contract?Initially, this was the plan - everything required for communication from host to device would be included in the contract. This adds a bit of reliability and predcitability when you are working with a single device. However, this has a few serious drawbacks. The first is that contracts tied to a single device, and can't be shared between people working on the same type of hardware, or identical clones of the same piece of hardware. The second issue is that it wouldn't be possible to run multiple devices on the same system using the same contract, which is a common use-case. The final issue is more fundamental - a contract should specify theinterfacebetween a host and device, and ideally is not specific to aparticularhost orparticulardevice. So long as they have the required libraries, it should work. For this reason, I decided the best option is to pass in the device identifier (serial number) at runtime. If one is not passed in, the auto-discovery mechanism will attempt to find a device.