Lincoln opened his speech by first indicating that he would not touch on "those matters of administration about which there is no special anxiety or excitement." The remainder of the speech would address the concerns of Southerners, who were apprehensive that "by the accession of a Republican Administration their property and their peace and personal security are to be endangered." Lincoln emphatically denied this assertion, and invited his listeners to consider his past speeches on the subject of slavery, together with the platform adopted by the Republican Party, which explicitly guaranteed the right of each individual state to decide for itself on the subject of slavery, together with the right of each state to be free from coercion of any kind from other states, or the Federal government. He went on to address several other points of particular interest at the time:
Slavery: Lincoln stated emphatically that he had "...no purpose, directly or indirectly, to interfere with the institution of slavery in the States where it exists. I believe I have no lawful right to do so, and I have no inclination to do so."
Legal status of the South: He asserted that as he had just taken an oath "to preserve, protect, and defend the United States Constitution", this oath enjoined him to see that the laws of the Union were faithfully executed in all statesâincluding those that had seceded.
Use of force: Lincoln promised that there would be no use of force against the South, unless it proved necessary for him to fulfill his obligation to "hold, occupy, and possess the property and places" belonging to the federal government, and to collect legal duties and imposts. However, if the South chose to actively take up arms against the Government, their insurrection would meet a firm and forceful response.
Secession: Referring to words in the preamble to the Constitution, Lincoln stated that the Constitution was established "to form a more perfect union" than the Articles of Confederation and Perpetual Union had effected. Since the Union established under the Articles was explicitly perpetual in name and text, thus the Union under the Constitution was equally perpetual. He added that even if the Constitution were to be construed as a simple contract, it could not be legally rescinded without an agreement between all parties, meaning all of the states, North and South.
Protection of slavery: Lincoln explicitly stated that he had no objection to the proposed Corwin Amendment to the Constitution, which had already been approved by both houses of the United States Congress. This amendment would have formally protected slavery in those states in which it already existed, and assured to each state the right to establish or repudiate it. Lincoln indicated that he thought that this right was already protected in the original Constitution, and thus that the Corwin Amendment merely reiterated what it already contained.
Slavery in the Territories: Lincoln asserted that nothing in the Constitution expressly said what either could or could not be done regarding slavery in the territories. He indicated his willingness to enforce the Fugitive Slave Act, so long as free blacks could be protected from being kidnapped and illegally sold into slavery through its misuse.
The postal service: The U.S. Mails would continue to operate throughout the South, "unless repelled."
Federal offices in the South: With no professional civil service in operation during this period of American history, Lincoln promised that he would not use the spoils system to appoint Northern office-holders to federal offices, such as postmasterships, located in the Southern states. Instead, he said he would "forego the use of such offices" rather than force "obnoxious strangers" upon the South.
Lincoln concluded his speech with an eloquent plea for calm and cool deliberation in the face of mounting tension throughout the nation. He assured the rebellious states that the Federal government would never initiate any conflict with them, and indicated his own conviction that once "touched" once more by "the better angels of our nature," the "mystic chords of memory" North and South would "yet swell the chorus of the Union."

Question: What kind of worries did Lincoln hold about the Fugitive Slave Act?
Answer: Free blacks being kidnapped and sold back into slavery.