Weber enlisted to fight in the Civil War as a colonel on May 16, 1861. He raised a German-American unit known as the "Turner Rifles," a company that eventually became a part of the 20th New York Infantry. Weber was stationed at Fort Monroe in Virginia. He took part in the capture of Fort Hatteras. From September 1861 until May 1862, he commanded Camp Hamilton, near Fort Monroe, being commissioned brigadier general of volunteers on April 28, 1862. He was at Newport News during the fight between the "Monitor" and "Virginia" in anticipation of a Confederate attack by land. He took part in the capture of Norfolk, Virginia in May, and then commanded at Suffolk until September, when he was ordered to the Army of the Potomac where he commanded the Third Brigade, Third Division, Second Army Corps in the Army of the Potomac.