In 1850, Colley went to California's gold fields, but returned the next year. In 1851, he served as was one of the judges of the plowing contest at the first Wisconsin State Fair. In 1853, he was again elected to the Assembly, this time as a member of the new Republican Party to succeed Whig William D. Murray, winning with a majority of 224 votes over former Democratic Assemblyman John Hackett. He was re-elected in 1854; and was succeeded after the 1855 election by fellow Republican Horatio J. Murray. In 1855, he was elected once more to the Rock County Board, this time from the City of Beloit; and again in 1859; in the latter year he was chairman of the board, as his father had been in 1847 and 1848. At some point during this time, he was also one of the three-person board elected by the county board as Superintendents of the Poor, in charge of the county poor house and poor farm.