Exhibit 10.4

RESIGNATION AND CONSULTANCY AGREEMENT

This resignation and consultancy agreement (the
Agreement) is entered into this 22nd day of January 2007 by and among Partners Bank, a federal savings bank (the Bank), Partners Financial Corporation (PFC) and James S. Weaver, President and Chief Executive
Officer of the Bank and PFC (Executive).

Whereas, the Bank has entered into an employment agreement with Executive dated
March 23, 2005 (Executive Employment Agreement); and

Whereas, PFC has guaranteed the obligations of the Bank under the
Executive Employment Agreement; and

Whereas Executive, the Bank and PFC believe it is in their best interests to modify the Executive
Employment Agreement on the terms and conditions set forth herein;

NOW, THEREFORE, in consideration of the recitals set forth above and
the mutual promises contained herein, the Executive, the Bank and PFC agree as follows:

1. Resignations. Effective on the date
hereof and on the terms and conditions set forth herein, Executive hereby resigns his responsibilities as (i) president and chief executive officer of the Bank and PFC and (ii) as a member of the Board of Directors of the Bank and PFC. The
Bank and PFC, respectively, accept Executives resignation as their respective president and chief executive officer. The Bank and PFC acknowledge Executives resignation from their respective Boards of Directors.

2. Termination of Executive Employment Agreement. The Executive Employment Agreement is terminated as of the date hereof, except for section 5
thereof which shall continue in effect.

3. Non-Solicitation. For six months from the date hereof (through July 22, 2007),
Executive will not contact or specifically solicit, whether personally or indirectly through agents or representatives, (a) any employee of the Bank or PFC for the purposes of terminating his or her employment with the Bank or PFC or
(b) any current customer of the Bank for the purposes of soliciting, diverting, or taking away or attempting to solicit, divert or take away from the bank, the banking business of any current customer of the Bank.

4. Consultant. For six months from the date hereof ( through July 22, 2007), Executive will make himself reasonably available to consult with
the Chairman of the Board of Directors or others he may designate, the executive officers of the Bank and PFC and officials of the Office of Thrift Supervision about the implementation of the business plan of the Bank and PFC; provided,
however, that Executives role as a consultant to the Bank and PFC shall end (a) if Executive is employed by another financial institution or (b) if he is employed in any capacity by any other employer and is not available to
provide advise to PFC and the Bank in person during the normal business hours of the Bank of (c) if he violates section 3 or section 8 hereof.



---

5. Payment. For the services rendered under section 4 hereof, Executive shall receive a salary of
$5,766.67 paid in bi-weekly installments, the last payment being made on or about July 22, 2007. No payment shall be made after Executives role as a consultant ceases before July 22, 2007 as provided by section 4 hereof.

6. Insurance. Executive shall continue to be provided a term life insurance policy through July 22, 2007 with the premiums for such policy
continuing to be paid by the Bank and/or PFC, after which date payments on behalf of the Executive shall cease. The Bank and/or PFC shall continue to pay its portion of the medical, dental and long-term disability coverage provided to employees of
the Bank for as long as Executive is eligible to participate in such insurance plans through July 22, 2007, after which date payments on behalf of the Executive shall cease. Commencing, however, on the date that Executive is no longer eligible
to participate in the medical, dental and long-term disability coverage provided to employees of the Bank through July 22, 2006, the Bank and/or PFC shall pay Executive an additional $333.23 bi-weekly and Executive acknowledges that after such
date, he shall be responsible for securing medical, dental and long-term disability coverage for himself.

7. Mutual Releases.
Executive, for himself, his attorneys, heirs, executors, administrators, successors and assigns releases forever the Bank and PFC and their respective past, present and future directors, officers, employees, agents, attorneys, successors and assigns
from any and all claims or causes of action Executive ever had, now has or hereafter can, shall or may have, known or unknown or unforeseen, in connection with the obligations of the Bank and PFC to Executive under the Executive Employment
Agreement. Executive further agrees not to sue the Bank and PFC and their respective past, present or and future directors, officers, employees, agents, attorneys, successors and assigns with respect to any claims he ever had, now has or hereafter
can, shall or may have, known, unknown or unforeseen, in connection with the obligations of the Bank and PFC to Executive under the Executive Employment Agreement.

The Bank and PFC for itself, respectively, and their respective past, present and future directors, officers, employees, agents, attorneys, successors and assigns release forever Executive, his attorneys, heirs,
executors, administrators, successors and assigns from any and all claims or causes of action the Bank and PFC ever had, now have or hereafter can, shall or may have, known, unknown or unforeseen, in connection with the obligations of Executive to
the Bank and PFC under the Executive Employment Agreement, except for any violations of the provisions of section 5 thereof which occur after the date hereof. The Bank and PFC further agree, respectively, not to sue the Executive, his attorneys,
heirs, executors, administrators, successors and assigns in connection with any claims they ever had, now have or hereafter can, shall or may have, known, unknown or unforeseen, in connection with the obligations of the Executive to the Bank and PFC
under the Executive Employment Agreement.

8. Public Announcement. Executive, the Bank and PFC agree not to make any public
announcement with regard to the subject matter of this Agreement without the consent of the other parties thereto, which consent shall not be unreasonably withheld.



---

9. Other. The parties hereto agree that this Agreement constitutes the entire agreement among them
with regard to the subject matter thereof. All parties agree that this Agreement and any disputes arising thereunder shall be governed by Florida law. The invalidity of any provision of this Agreement shall not affect any other provision of the
Agreement, which shall remain in full force and effect; nor shall the invalidity of a portion of any provision of this Agreement affect the balance of such provision.

IN WITNESS WHEREOF, this Agreement has been duly executed and delivered by the Executive and a duly authorized officer of the Bank and PFC on the date first written above.



|                                   
|-----------------------------------
|EXECUTIVE                          
|                                   
|/s/ James S. Weaver                
|James S. Weaver                    
|                                   
|PARTNERS BANK                      
|                                   
|/s/ John G. Wolf                   
|John C. Wolf, Chairman of the Board
|                                   
|PARTNERS FINANCIAL CORPORATION     
|                                   
|/s/ John G. Wolf                   
|John G. Wolf, Chairman of the Board