As is the case with any console, the Master System struggled for multiple reasons. The most pressing was the fact that, by the time the Master System (or Mark III as it was known in Japan) arrived, Nintendo's Famicom / NES had pretty much dominated the Japanese and North American markets. Nintendo also locked down publishers by making them enter agreements where their titles would be exclusive to the NES, preventing Sega from releasing them on the Master System.
Another major issue was the fact that the Master System was distributed by Tonka in North America, a company which lacked the experience to challenge Nintendo. Sega would eventually assume distribution duties in the region, but by that point, it was too late.
It's worth noting that the Master System was incredibly successful in Europe, where Nintendo had a weaker presence. It was also massively popular in Brazil, where it was distributed and marketed by Tec Toy.