A point guard, Brown attended Long Beach High School. He won a gold medal with Team USA in basketball at the 1961 Maccabiah Games.