A. Appendix
A.1. Dataset details
CARLA refuses to spawn agents that collide with theenvironment, including the ground. To ensure agents aregrounded, for any asset that causes a spawn collision, we
increase its Z coordinate and try to spawn again . This approach allows us to place every agent on the map, albeit
some of the conflicting agents have to ‘drop’ from above,and consequently we wait for 50 timesteps so those agentscan settle. In that duration, the autopilot policy guides theagents to satisfactory positions. After those 50 steps, wethen record for another 150 steps and save every 15th frame.
The resulting episodes each have ten frames following an
initial distribution influenced by Nuscenes and CARLA,
and a traffic policy influenced by only CARLA determining the final settled distribution.
We then need the 2D ground truth boxes for each asset.
We found the existing suggested approach lacking because
it frequently has trouble with occlusions and other challenging scenarios. See below for heuristics we developed to help
filter the ground truth boxes. While they are not airtight, theresulting ground truths were qualitatively perceived as morereliable.
• Filter Height: We require that the final 2d box is atleast 30 pixels. This is in between the easy (40) andmedium/hard (25) settings on KITTI [15].
• Max Distance: We require that the ground truth detection not be more than 250 meters away. We enforce
this through the use of a depth camera attached to theego agent.
• Visible Pixel Percent (VPP) and Min Visible Count(MVC): The 2D box is attained by pairing the 3D boxwith the camera’s calibration. With the latter, we getthe closest point P to the ego agent. We then get thedepth camera’s output at the 2D box. VPP asks whatpercent t of that box is closer than P and filters it ift 80, ensuring that at least 20% of the object is notoccluded. MVC asks how many pixels q are further
than P and filters it if q < 1300, ensuring that the occluded object is big enough.

A.2. Supporting charts

Figure 7: Charts showing increasing both data and model capacity
at the same time. The left side ranges over model capacity withmaximum IID data size (85000), while the right side ranges overIID data size with a bigger model - 34C4.

Figure 8: Performance of 18C4 on select test sets when adding mode data from the three bikes, the ColaCar, and the Cybertruck on top ofeither 10000 or 85000 base IID data. Towards improving the results, these two charts show that it is not the absolute count of the mode datathat is important but rather the percent of it relative to the IID data. We see that in how the trendlines for the two bases are only consistentin the percent chart. The other modes are not shown for clarity but it holds in general.

Figure 9: We can see that the model size does matter in that for every group the 34C4 model improves over the 18C4 model. However, the increase is quite small and the data quality and quantity
appear to matter much more.

Figure 10: Results adding mode data to the base IID 10000 training set. This is the same as Figure 5 but zoomed into just [0, 1000]. The
five modes in the top left are the Cybertruck, Cola Car, Diamondback, Gazelle, and Crossbike, each added in equal proportion.

Figure 11: Comparing rotation and weather results for MLM andRandom intervention strategies. We see that MLM fits with Original much better than Random does. Further, Random has a muchwider berth of possible problematic modes, which is a concern
given practical limits to model capacity and data budgets.