The surface of the town is greatly diversified by hills and valleys. Another range of hills extend through the eastern half, thus dividing the town into three sections, designated as the east, west and middle. The ever renowned Green Mountains lie upon the east, west of which is Otter Creek valley. A portion of this valley is enclosed within the limits of Danby. The eastern range of hills terminate upon the south, forming a pass.
Otter Creek flows through a small portion of the town, in the north-east.
The town is well watered by numerous streams, the two principal of which is Mill River and Flower Brook. Mill River is formed by the junction of a large number of small streams, one of which rises in the extreme south-western part of the town. Mill River flows an easterly course through the town, winding and twisting among the hills, until it empties into Otter Creek, in the town of Mt. Tabor. Flower Brook rises in the north-western part of the town, flows southerly for about one mile, then flows westerly and empties into Pawlet river, in the town of Pawlet. Danby Pond is situated near the center of the town, and its outlet flows into Mill River.