The main concern is the influx of illegal immigrants crossing the border, including migrant families and potentially even terrorists. The video highlights the worry about the federal government's response to this issue.