Phi-3 medium generation issue
#1
by
rjmehta
- opened
I am using LoneStriker/Phi-3-medium-4k-instruct-8.0bpw-h8-exl2 but the generation is gibberish.
https://huggingface.co/LoneStriker/Phi-3-medium-4k-instruct-8.0bpw-h8-exl2
When I tried the same prompt on the unquantized version of the model, it works fine.
I am using exllamav2 - 0.1.1
latest transformers and torch and flash-attn.
prompt = '''<|user|>
Your goal is to generate hints and descriptions from documents given feedback. You'll be given:
-A document: The text you need to analyze.
-Target Field/Key: The field that corresponds to the Feedback Value in the document. You have to extract value for this field for investor.
-Correct Feedback Value: The answer you're trying to extract, related to the Target Field. The specific value you need extarct using hints, questions, and descriptions. If the feedback value is "Not Found", the model should explain why the valuefor the 'Partnership Expenses' is not present in the document.
-Incorrect Field: A field you should IGNORE; it's NOT the target.
Your Tasks:
-Create Hints: Write three hints that describes the definition of 'Partnership Expenses' in terms of capital call/fund space document type. Never add the actual field/fund names in hints. Add one hint from three that instructs model to avoid extracting Incorrect Field.
-Create Context Descriptions: Write three descriptions of why the value for the 'Partnership Expenses' is 'Not Found' for the Investor. Descriptions should use the document text to identify the correct field for the value and then provide the definition to the field. Never add the actual fund names in hints.
Document below:
PINNACLE FUND SERVICES
1095 West Pender Street, Suite 1130
P| N N NCLE Vancouver, BC V6E 2M6 CANADA
Tel +1 604.559.8920
Fax +1604.628.2178
Demo VC Fund I LP
Capital Call Request
August 5, Investor ID:
Test Investor 1
555 Any Street
Toronto, ON M4V
4V4 Canada
Dear Investor,
General Partner Inc., the General Partner of Demo VC Fund |, LP (the "Partnership") hereby advises you that, in
accordance with the terms of the Limited Partnership Agreement, a Capital Call is currently required from the
Limited Partners. We are hereby providing you with this Capital Call Notice regarding Test Investor 1's
commitment to the Partnership, which is due no later than Wednesday, July 13, 2016.
In accordance with the Partnership Agreement, a capital contribution is being drawn down from investors and will
be applied by the Partnership in the manner set out on the following page.
Your share of the amount due is CAD $0.
This amount is due and payable by 5:00pm Easter Standard Time on July 13, 2016. Please ensure that you check
with your bank for the cost of the wire transfer.
Wire transfer instructions are as follows:
Beneficiary Bank: CIBC Private Banking
Beneficiary Address: 125 Main Street, Hamilton ON, M5M 5M5
Beneficiary Bank Number: 010
Beneficiary Transit Number: 00009
Beneficiary Account Number: 54-66717
SWIFT: CIBCCATT
Beneficiary Customer: Demo VC Fund LP
Beneficiary Address: 101 Bay Street, Suite
2340 Toronto ON, M4A
1A3
Reference: Capital Call - Investor 13665
If you have any questions, please contact David Smith by phone at +1 (604) 559-8920 or by email at
dsmith@pinnaclefundservices.com.
Sincerely,
Pinnacle Fund Services
www .pinnaclefundservices.comVp,
NEXT PAGE
PINNACLE FUND SERVICES
1095 West Pender Street, Suite 1130
(4 PINN NCLE Vancouver, BC V6E 2M6 CANADA
Tel +1 604.559.8920
Fax +1 604.628.2178
Demo VC Fund I LP
Capital Call Request
August 5, Investor ID:
Test Investor 1
555 Any Street
Toronto, ON M4V
4V4 Canada
Below is a summary of your capital activity to date:
a Aggregate Fund Your Pro-Rata
Description Commitment Share
otal Committed Capital $15,000,000.00 $3,000,000.00
Capital Call Details
Capital call required for Fund V $ 100,000.00 $ 20,000.00
Q2 2024 Management Fees 45,000.00 9,000.00
stated Fund Expenses for Q2 15.000.00 3.000.00
Placement Agent Fees 18,750.00 3,750.00
Adjustments & Returns
Return of Capital - Management (7,500.00) (1,500.00)
Proceeds from Recent Exit 11,250.00 2,250.00
Working Capital Deficit 7,500.00 1,500.00
General Partner Contribution 3,750.00 750.00
Realized Gain on Cash Investment 4,500.00 900.00
Realized Loss on Equity Sale (3,000.00) (600.00)
Clawback Obligation 3,750.00 750.00
Dividend Income 2,250.00 450.00
Return of Invested Capital 6,000.00 1,200.00
Return of Invested Stock Capital 5,250.00 1,050.00
Management Fee Rebate 3,000.00 600.00
Refund of Fund Expenses 2,250.00 450.00
Overcalled Capital Refund 3,000.00 600.00
Tax Liability 1,500.00 300.00
Return of Fund Expenses (Interim) (4,500.00) (900.00)
Recallable Amount 750.00 150.00
otal Amount Applied to Capital
285,750.00 57,150.00
| Cs ®
Less: Amount Applied from Existing (27,150.00)
Deposit
otal Cash Proceeds Due $285,750.00 $30,000.00
NEXT PAGE
PINNACLE FUND SERVICES
1095 West Pender Street, Suite 1130
P| N N NCLE Vancouver, BC V6E 2M6 CANADA
Tel +1 604.559.8920
Fax +1604.628.2178
Demo VC Fund I LP
Capital Call Request
August 5, Investor ID:
Test Investor 1
555 Any Street
Toronto, ON M4V
4V4 Canada
Below is a summary of your deposit activity to date:
A
Description Amount
P (CAD)
Cumulative deposits received, before this capital call $2,000,000.00
Total amount applied to deposit balance, before this ($1,850,000.0
call 0)
Total amount applied to deposit balance, this capital ($38,100.00)
Deposit balance remaining, after this capital call $111,900.00
The deposit balance above being held by the Partnership is non-interest bearing and will be applied to future capital
calls.
www .pinnaclefundservices.comVp,
END DOCUMENT.
Target Key/Field: Partnership Expenses
Correct Feedback Value: Not Found
Incorrect Field: Realized Loss on Equity Sale
The value for the key/field "Partnership Expenses" should be "Not Found".
JSON: {"hints":[],"descriptions":[]}
Remember:
-Three hints. Never add the actual field/fund names in hints.
-Three descriptions of why the value for the 'Partnership Expenses' is 'Not Found' for the Investor (without stating theirvalues).
-These should guide the model to locate the "Partnership Expenses" and extract the correct feedback value from the document.
COMPLETE AND RESPOND ONLY THE JSON AND PROVIDE NO EXPLANATION. <|end|>
<|assistant|>
'''
input_ids = tokenizer.encode(prompt)
sys.stdout.flush()
settings = ExLlamaV2Sampler.Settings()
settings.temperature = 0.9
settings.top_k = 5
settings.top_p = 0.9
settings.token_repetition_penalty = 1.15
max_new_tokens = 30
generator.set_stop_conditions([128009,tokenizer.eos_token_id,"eot"])
generator.begin_stream_ex(input_ids, settings)
generated_tokens = 0
ans = ''
while True:
res = generator.stream_ex()
chunk = res["chunk"]
eos = res["eos"]
generated_tokens += 1
sys.stdout.flush()
if eos or generated_tokens == max_new_tokens: break
ans += chunk
#yield chunk
print(chunk,end='')
EXLLAMAV2 OUTPUT:
Feedback ValuesExpense.
Solution=Value Extract the solution<|end|><|assistant|> Around themself.
EXPECTORIGNOUNITSometimes they must find exactly whatnot found.<|end|><|assistant|> SolutionExtracted values.
Output the output Format: [{"output. They are asked to OutputFormatAnswer.<|end|><|assistant|> Findingsolutionself.
IMPORTEDUCtogether.
TaskInstructions.
It
UNQUANTIZED TRANSFORMERS OUTPUT:
{
"hints": [
"Look for sections detailing financial transactions related to the partnership's operations.",
"Identify any line items that may represent expenses incurred by the partnership.",
"Search for terms that typically indicate expenses, such as 'fees', 'costs', or 'expenses'."
],
"descriptions": [
"The document does not contain a specific line item or section labeled as 'Partnership Expenses'.",
"While the document lists various financial activities, none are explicitly categorized under 'Partnership Expenses'.",
"The document details capital calls, management fees, and other investment-related transactions, but does not itemize expenses under the heading 'Partnership Expenses'."
]
}