## 3 The Bike-Share Case
### Evaluation--Partial Model Testing  
The objective of partial model testing, as suggested by [PERSON], is to broaden the perspective from the task level--typically the initial focus of the intervention--to encompass its immediate process context before fully integrating it into the holistic model. For more traditional policy changes that do not involve machine-learning agents, [PERSON] advocated for partial testing to prevent what he called the \"leap of logic\" from equations to consequences [58]. This approach assumes that, under bounded rationality, policy interventions are generally reasonable at the local or subsystem level before accounting for the dynamic complexity of the entire system. Partial testing, therefore, helps confirm that an intervention is rational in intention before expanding its scope. Since machine learning introduces new dynamics, particularly in navigating decision-making alongside humans, partial testing becomes even more critical. In the bike-share experiment, we apply this approach to test the reasonableness of using a deep learning method for demand forecasting instead of the traditional exponential smoothing technique--a well-established statistical procedure [59]. While the forecast accuracy metric showed substantial improvement with the RNN model, which intuitively makes sense given its ability to leverage correlates of demand that a univariate method like exponential smoothing cannot, we broaden the testing scope and use a process-oriented metric to determine if these improvements hold at this level. The following steps outline the procedure for executing the partial tests. **Inputs:**  
ML (LSTM) and exponential smoothing (STAT) predictions for a given bike station/day. Maximum capacity of a station. **Procedure:**  
* Calculation of Just-in-Time Inventory Balancing (Receipts and Issues):
* Run the ML and STAT predictions through a simplified stocks and flows structure to compute projected stocks. * Whenever an outage occurs (either lack of bikes or docks), record the volume as a planned balancing receipt or issue. * Calculation of the Process Metric: Total Outages with Just-in-Time Balancing (ML vs. STAT Predictions):
* Run actual or simulated rental and return demands through a structure that models actual stock (aligned with the projected stock structure). * Use planned receipts and issues (from the previous step) as lookups to adjust rentals and returns. * Record the bikes and docks outages. Figure 12 depicts the structure of the stocks and flows used to compute the process metric. Despite its simplicity, the partial test helps determine whether individual rationality--in this case, the superior forecast accuracy of the RNN's rental and return predictions--translates to collective rationality. The latter refers to whether the combined use of these predictions in a more realistic setting continues to outperform the simpler alternative. In our first iteration, the performance of several stations under ML was unexpectedly worse. The charts in Figure 13 illustrate a typical outcome. The blue and red lines in the stock outages chart represent the process metric for the station Broadway & E 21 St from 22-25 August 2023, for STAT and ML, respectively. Despite ML's forecast accuracy (meansured by Mean Squared Error) being about 28% better for rentals and 19% for returns, its process metric is 95% worse, with 72 units versus 37 units of combined bike and dock outages. A closer inspection, tracing the red and blue arrows, reveals the reason by comparing the projected and actual utilisation over time. The projected utilisation, defined as the ratio of projected stock to maximum capacity, is consistently and significantly lower for ML than the actual utilisation, indicating a systematic bias in the predictions. Specifically, this involves either under-forecasting returns or over-forecasting rentals or both. In this case, returns were severely under-forecasted, leading to just-in-time receipts that caused dock outages (as receipts exceeded maximum capacity).