I'm handling this error which appears in one of our system builder's jenkins pipelines:

<error>
.../linux/x86_64/usr/bin:/opt/qos224/host/linux/x86_64/usr/bin:/usr/bin/cmake/bin:/opt/tttech/TTE/TTEbuild/5.6.2002:/opt/tttech/TTE/TTEplan/5.6.2000:/opt/tttech/TTE/TTEload/5.6.2000:/opt/tttech/TTE/TTEutilities/5.6.2000:/opt/tttech/TTE/TTEworkbench/5.6.2000:/opt/cpptest:/opt/cpptest/bin'
env['QACPATH'] = ''C:\PRQA\PRQA-Framework-2.2.2\common\bin''
env['QNX_HOST'] = '/opt/qos22/host/linux/x86_64'
env['QNX_TARGET'] = '/opt/qos22/target/qnx7'
qacli not found
scons: done reading SConscript files.
scons: Building targets ...
Making base/Rte_TTTech_CpApBISTASIL_SH00/GHv7_1_6_202014_s32g/Rte_Modes_CpApBISTASIL_SH00.o ...
/usr/ghs/comp_202014/ccarm -c -MMD -DKL5_DEFAULT_ON -cpu=cortexm7 -C99 -Osize -Wundef -Wimplicit-int -Wshadow -Wtrigraphs --prototype_errors -preprocess_assembly_files -nostartfile -keeptempfiles -farcalls -no_misalign_pack -fsingle -list -globalcheck=normal -Wformat --ghstd=last --incorrect_pragma_errors --unknown_pragma_errors -dual_debug -fpu=vfpv5_d16 --no_vla -Wconvert64 --gnu_asm -lnk=-v -G -dwarf2 -ga -D_UTC_Y=2015 -D_UTC_MM=1 -D_UTC_DD=1 -D_UTC_H=0 -D_UTC_M=0 -D_UTC_S=0 -D_UTC_STR=20150101000000 -DRTE_E2EP_ERROR_INJECTION -DRTE_SEND_EMPTY_FRAMES -DBC_RWS -DBC_EH_MASTER -DBC_OS_AUTOSAR -DBC_OS_MICROSAR -DBC_RTE_E2EP_ERROR_INJECTION -DBC_SOA_EH -DBC_TASK_MONITORING -DBC_TRACE -DSWC_NAME=CtApBISTASIL_SH00 -DSWC_ID=SWCID_CtApBISTASIL_SH00 -D_OS_AUTOSAR_ -D_GHS_C_ARM_ -DTRACE_ENABLE=0 -D_WATCHDOG_DISABLE -DSKIP_MAGIC_NUMBER -DPFSW_BUILD_OS_AUTOSAR -DTRACE_RUNNABLE_COMPILE_ENABLE -DBC_PROFILING_PERFMEAS -DBC_PROFILING_PERFMEASTH -DBC_PROFILING_REMTIME -DBC_PROFILING_GLIWA_T1 -DBC_S32G_GLIWA_ENABLED -DDEBUG -DS32G2XX -Ds32xx -D_CPU_S32G274A_ -D_SOC_S32_ -D_HOST_SH00_ -DMW_HOST_DST=MW_DST_ADAS1_B1_EXT_SH00 -D_SOA_PLATFORM_SH_ -DENABLE_COMMUNICATION_TEST -DADAS_MIDDLEWARE_TEST -DTEST_ITF -DTEST_BUILD @base/Rte_TTTech_CpApBISTASIL_SH00/GHv7_1_6_202014_s32g/Rte_Modes_CpApBISTASIL_SH00.ccdat /data/jenkins_b3/DLaGrY/1700-Configuration/RDB2/1700-generated-config/ADAS1_EXT/rte/SH00/Rte_TTTech/Rte_CpApBISTASIL_SH00/Rte_Modes_CpApBISTASIL_SH00.c -o base/Rte_TTTech_CpApBISTASIL_SH00/GHv7_1_6_202014_s32g/Rte_Modes_CpApBISTASIL_SH00.o
"/data/jenkins_b3/DLaGrY/1700-Configuration/RDB2/1700-generated-config/ADAS1_EXT/rte/SH00/Rte_TTTech/Rte_CpApBISTASIL_SH00/Rte_Modes_CpApBISTASIL_SH00.c", line 59: fatal error #2330:
         cannot open source file "Rte_CpCdTimeMonitorMaster00_SH00_Internal.h"
 #include "Rte_CpCdTimeMonitorMaster00_SH00_Internal.h"
                                                       ^

scons: *** [base/Rte_TTTech_CpApBISTASIL_SH00/GHv7_1_6_202014_s32g/Rte_Modes_CpApBISTASIL_SH00.o] Error 1
scons: building terminated because of errors.

scons: *** [/data/jenkins_b3/DLaGrY/forge_targets/mxmlDebugReleaseFolder_-_TestITFADAS1_EXTADAS1_B1_EXTSH00] Error 2
scons: building terminated because of errors.
Took 773.48 seconds to execute
Wrote /data/jenkins_b3/DLaGrY/upload_to_artifactory/execution_log.json
Wrote execution_log_legacy.json 
</error>

let me put you up to date:

1. this is the only builder that fails, related to the "ADAS" product. the ADAS product builds different configurations for two systems: ADAS1 and ADAS1_EXT: only ADAS1_EXT fails, as seen in the error above
2. the issue is regarding a header file which is indeed generated
3. looking at the builder logs, we can confirm that this header is generated (locally too)
4. **locally the build works fine, doing the same steps as on the pipeline,  so the challenge is to understand why does it fail only in the pipeline**
5. we also found that `1700-Configuration/RDB2/1700-generated-config/ADAS1_EXT/rte/SH00/Rte_TTTech/Rte_CpApBISTASIL_SH00/Rte_Modes_CpApBISTASIL_SH00.c` doesn't exist locally, but seems to exist in the pipeline
6. there is an script which directly deletes this file, and this script indeed does execute and apparently, removes the file, these are the logs from the builders

<remove_builder_log>
[2024-10-04 00:26:59.534] [.temp.remove_MCF_file         ] [INFO               ] - REMOVED out: 1700-Configuration/RDB2/1700-generated-config/ADAS1_EXT/rte/SH00/Rte_TTTech/Rte_CpApBISTASIL_SH00/Rte_Modes_CpApBISTASIL_SH00.c
</remove_builder_log>

so in summary: we don't know why this .c file even exists in the builders environment. the code generators seem to be working in the same way as locally (acording to the logs). we can't access the builders env but we can access the logs which contain mostly all operations executed there. also, there have been some changes in the generators code recently, but nothing directly related to this specific file (this file is generated by a different generator which wasn't changed, called "gen_rte_modes")

more context:
<generator_call>
<wish name="gen_rte_modes">
            <description>
                <p>name: Generate the RTE Modes source (.c) file</p>
                <p>Responsible: AD_PF_iCOM_Iberia_A</p>
            </description>
            <properties>
                <property key="SC-Step" />
                <property key="Service">MotionWise Com RTE Service</property>
                <property key="Stage">OS and RTE Configuration</property>
                <property key="Process">Configuration</property>
                <property key="Step">Generation of the RTE Modes</property>
                <property key="Execution_Scope">ecu</property>
                <property key="Workflow">all</property>
            </properties>
            <tool engine="motionwise_code_factory">
                <arg>--genplugin {CAMEL:motionwise_com_rte}/RTE_Modes/RteModesPlugin.py</arg>
                <arg>--template_path {CAMEL:motionwise_com_rte}/RTE_Modes/templates/Rte_Modes_SWCP.c.j2</arg>
                <arg>--outpath {OUTPUT:mcl:1700_Rte_Modes_SWCP#system/{PARAM:ECU_name}}</arg>
            </tool>
            <post_actions>
                <!-- This post-action should be removed when MCF enables SWC filtering (to skip non-generated ones) -->
                <tool engine="python" environment="3">
                    <arg>{CAMEL:com_data_model}/COM_Utils/.temp/remove_MCF_file.py</arg>
                    <arg>--genie gen_rte_modes</arg>
                    <arg>--outpath {OUTPUT:mcl:1700_Rte_Modes_SWCP#system/{PARAM:ECU_name}}</arg>
                </tool>
                <tool engine="manual">
                    <arg>{INPUT:mxml:1900_-_Core_-_MOM_skeleton}</arg>
                    <arg>{OUTPUT:mcl:1700_Rte_Modes_SWCP#system/{PARAM:ECU_name}}</arg>
                </tool>
            </post_actions>
        </wish>
</generator_call>