In spring 1941, he was drafted again and assigned to "Einsatzkommando" 3, part of the "Einsatzgruppe" stationed in Bad Düben. In October 1941, he was assigned to work in the Office of the Commander of Security Police ("Sicherheitsdienst" or SD) and Security Police ("Sicherheitspolizei" or Sipo) in Vilnius, then part of the Reichskommissariat Ostland. Weiss held this position until July 1944. He was responsible for all aspects of the repression against the Jewish population of Vilnius, which is estimated to have been around 50,000 during the Holocaust. Despite his low rank of technical sergeant ("SS-Hauptscharführer"), he was in charge of the Vilna Ghetto and nearby Lukiškės Prison, as well as commanded the "Ypatingasis būrys" killing squad responsible for the Ponary massacre until 1943. He personally supervised 13 to 15 executions at the site. In July 1943, Weiss became chief of the Gestapo prison in Vilnius in occupied Lithuania. In September 1943, he was selected to coordinate the work of the Sonderkommando 1005 to erase evidence of Jewish exterminations i.e unearthing and burning of the corpses.