introvoyz041/FactoryBench_Generation huggingface dataset:
Question type: troubleshooting
Question template: Given the sensor stream below, does the machine show signs of anomalous behavior? If yes, identify the most likely root cause and describe the steps you would take to fix it.
Answer format: free_form
Notes: Expected answer covers three parts: (1) anomaly present or not, (2) root cause identification if anomalous, (3) ordered remediation steps. Scalable via troubleshooting instruction databases and root-cause-to-action mappings. Evaluated with LLM-as-Judge against a reference answer derived from the root cause and anomaly labels.
Question type: optimization
Question template: An engineer wants to increase the effectiveness and accuracy of this machine. Based on the sensor stream below, what operational changes or parameter adjustments could help achieve this? Do not justify your answer, simply indicate steps to take (or specify if nothing can be done).
Answer format: free_form
Notes: Expected answer identifies sub-optimal operating conditions visible in the sensor stream and proposes actionable adjustments (e.g., speed, payload, trajectory tuning). Scalable by recording episodes with deliberately sub-optimal parameters alongside their optimized counterparts, using the delta as a reference. Evaluated with LLM-as-Judge against a reference derived from the known parameter gap.
Question type: optimization
Question template: The 4 sensor streams below each correspond to a complete task execution padded to a common length. Consider only the task completion portion of each stream (up to the end of the final task phase) when assessing duration. Rank them from shortest to longest task duration. Answer only with a four letter string indicating your ranking (e.g. DCAB), nothing else.
Answer format: ranking
Notes: Model must output a permutation of the episode labels (A, B, ..., N). Ground truth ranking is derived from the episode-level duration metadata field recorded during trajectory optimization blocks. Evaluate by comparing the predicted permutation to the ground truth; accept exact match or compute Kendall's tau for partial credit.
Question type: optimization
Question template: The 4 sensor streams below each correspond to a complete task execution padded to a common length. Consider only the task completion portion of each stream (up to the end of the final task phase) when assessing energy consumption. Rank them from most to least energy-efficient. Answer only with a four letter string indicating your ranking (e.g. DCAB), nothing else.
Answer format: ranking
Notes: Model must output a permutation of the episode labels (A, B, ..., N). Ground truth ranking is derived from the episode-level energy metadata field (total joint energy in joules) recorded during trajectory optimization blocks. Evaluate by comparing the predicted permutation to the ground truth; accept exact match or compute Kendall's tau for partial credit.