The best colors to wear with brown clothes are white, black, navy, orange, and olive green. These colors will help to complement the brown color of the clothes and will help to create a cohesive outfit.