Chatbot Tutorial problem

715456786b674e5bf49e: <p>Hi,</p>
<p>I’m new to pytorch and have been following the many tutorials available.<br>
But, When I did The CHATBOT TUTORIAL<br>
(<a href="https://pytorch.org/tutorials/beginner/chatbot_tutorial.html?highlight=chatbot%20tutorial" class="inline-onebox" rel="noopener nofollow ugc">Chatbot Tutorial — PyTorch Tutorials 2.2.0+cu121 documentation</a>) is not work.</p>
<p>Like the figure below<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/8/83c9d2dd5dd04aebe3c39cf33560273dfefe8966.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/83c9d2dd5dd04aebe3c39cf33560273dfefe8966" title="8788"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/8/83c9d2dd5dd04aebe3c39cf33560273dfefe8966_2_580x499.jpeg" alt="8788" data-base62-sha1="iNQX8KWkRX0ZmyMaUAopEcyPrXE" width="580" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/8/83c9d2dd5dd04aebe3c39cf33560273dfefe8966_2_580x499.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/8/83c9d2dd5dd04aebe3c39cf33560273dfefe8966_2_870x748.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/8/83c9d2dd5dd04aebe3c39cf33560273dfefe8966.jpeg 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">8788</span><span class="informations">893×769 109 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What should I do and what is causing this?</p>
<p>USE_CUDA = torch.cuda.is_available()<br>
device = torch.device(“cuda” if USE_CUDA else “cpu”)</p>
<p>corpus_name = “cornell movie-dialogs corpus”<br>
corpus = os.path.join(“data”, corpus_name)</p>
<p>def printLines(file, n=10):<br>
with open(file, ‘rb’) as datafile:<br>
lines = datafile.readlines()<br>
for line in lines[:n]:<br>
print(line)</p>
<p><span class="hashtag-raw">#printLines</span>(os.path.join(corpus, “movie_lines.txt”))</p>
<h1><a name="splits-each-line-of-the-file-into-a-dictionary-of-fields-1" class="anchor" href="#splits-each-line-of-the-file-into-a-dictionary-of-fields-1"></a>Splits each line of the file into a dictionary of fields</h1>
<p>def loadLines(fileName, fields):<br>
lines = {}<br>
with open(fileName, ‘r’, encoding=‘iso-8859-1’) as f:<br>
for line in f:<br>
values = line.split(" +++$+++ ")<br>
# Extract fields<br>
lineObj = {}<br>
for i, field in enumerate(fields):<br>
lineObj[field] = values[i]<br>
lines[lineObj[‘lineID’]] = lineObj<br>
return lines</p>
<h1><a name="groups-fields-of-lines-from-loadlines-into-conversations-based-on-movie_conversationstxt-2" class="anchor" href="#groups-fields-of-lines-from-loadlines-into-conversations-based-on-movie_conversationstxt-2"></a>Groups fields of lines from <code>loadLines</code> into conversations based on <em>movie_conversations.txt</em></h1>
<p>def loadConversations(fileName, lines, fields):<br>
conversations = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
with open(fileName, ‘r’, encoding=‘iso-8859-1’) as f:<br>
for line in f:<br>
values = line.split(" +++$+++ ")<br>
# Extract fields<br>
convObj = {}<br>
for i, field in enumerate(fields):<br>
convObj[field] = values[i]<br>
# Convert string to list (convObj[“utteranceIDs”] == “[‘L598485’, ‘L598486’, …]”)<br>
lineIds = eval(convObj[“utteranceIDs”])<br>
# Reassemble lines<br>
convObj[“lines”] = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for lineId in lineIds:<br>
convObj[“lines”].append(lines[lineId])<br>
conversations.append(convObj)<br>
return conversations</p>
<h1><a name="extracts-pairs-of-sentences-from-conversations-3" class="anchor" href="#extracts-pairs-of-sentences-from-conversations-3"></a>Extracts pairs of sentences from conversations</h1>
<p>def extractSentencePairs(conversations):<br>
qa_pairs = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for conversation in conversations:<br>
# Iterate over all the lines of the conversation<br>
for i in range(len(conversation[“lines”]) - 1):  # We ignore the last line (no answer for it)<br>
inputLine = conversation[“lines”][i][“text”].strip()<br>
targetLine = conversation[“lines”][i+1][“text”].strip()<br>
# Filter wrong samples (if one of the lists is empty)<br>
if inputLine and targetLine:<br>
qa_pairs.append([inputLine, targetLine])<br>
return qa_pairs</p>
<p>datafile = os.path.join(corpus, “formatted_movie_lines.txt”)</p>
<p>delimiter = ‘\t’</p>
<h1><a name="unescape-the-delimiter-4" class="anchor" href="#unescape-the-delimiter-4"></a>Unescape the delimiter</h1>
<p>delimiter = str(codecs.decode(delimiter, “unicode_escape”))</p>
<p>lines = {}<br>
conversations = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
MOVIE_LINES_FIELDS = [“lineID”, “characterID”, “movieID”, “character”, “text”]<br>
MOVIE_CONVERSATIONS_FIELDS = [“character1ID”, “character2ID”, “movieID”, “utteranceIDs”]</p>
<h1><a name="load-lines-and-process-conversations-5" class="anchor" href="#load-lines-and-process-conversations-5"></a>Load lines and process conversations</h1>
<p>print(“\nProcessing corpus…”)<br>
lines = loadLines(os.path.join(corpus, “movie_lines.txt”), MOVIE_LINES_FIELDS)<br>
print(“\nLoading conversations…”)<br>
conversations = loadConversations(os.path.join(corpus, “movie_conversations.txt”),<br>
lines, MOVIE_CONVERSATIONS_FIELDS)</p>
<h1><a name="write-new-csv-file-6" class="anchor" href="#write-new-csv-file-6"></a>Write new csv file</h1>
<p>print(“\nWriting newly formatted file…”)<br>
with open(datafile, ‘w’, encoding=‘utf-8’) as outputfile:<br>
writer = csv.writer(outputfile, delimiter=delimiter)<br>
for pair in extractSentencePairs(conversations):<br>
writer.writerow(pair)</p>
<h1><a name="print-a-sample-of-lines-7" class="anchor" href="#print-a-sample-of-lines-7"></a>Print a sample of lines</h1>
<p>print(“\nSample lines from file:”)<br>
printLines(datafile)</p>
<h1><a name="default-word-tokens-8" class="anchor" href="#default-word-tokens-8"></a>Default word tokens</h1>
<p>PAD_token = 0  # Used for padding short sentences<br>
SOS_token = 1  # Start-of-sentence token<br>
EOS_token = 2  # End-of-sentence token</p>
<p>class Voc:<br>
def <strong>init</strong>(self, name):<br>
self.name = name<br>
self.trimmed = False<br>
self.word2index = {}<br>
self.word2count = {}<br>
self.index2word = {PAD_token: “PAD”, SOS_token: “SOS”, EOS_token: “EOS”}<br>
self.num_words = 3  # Count SOS, EOS, PAD</p>
<pre><code>def addSentence(self, sentence):
    for word in sentence.split(' '):
        self.addWord(word)

def addWord(self, word):
    if word not in self.word2index:
        self.word2index[word] = self.num_words
        self.word2count[word] = 1
        self.index2word[self.num_words] = word
        self.num_words += 1
    else:
        self.word2count[word] += 1

# Remove words below a certain count threshold
def trim(self, min_count):
    if self.trimmed:
        return
    self.trimmed = True

    keep_words = []

    for k, v in self.word2count.items():
        if v &gt;= min_count:
            keep_words.append(k)

    print('keep_words {} / {} = {:.4f}'.format(
        len(keep_words), len(self.word2index), len(keep_words) / len(self.word2index)
    ))

    # Reinitialize dictionaries
    self.word2index = {}
    self.word2count = {}
    self.index2word = {PAD_token: "PAD", SOS_token: "SOS", EOS_token: "EOS"}
    self.num_words = 3 # Count default tokens

    for word in keep_words:
        self.addWord(word)
</code></pre>
<p>MAX_LENGTH = 10  # Maximum sentence length to consider</p>
<h1><a name="turn-a-unicode-string-to-plain-ascii-thanks-to-9" class="anchor" href="#turn-a-unicode-string-to-plain-ascii-thanks-to-9"></a>Turn a Unicode string to plain ASCII, thanks to</h1>
<h1><a name="httpstackoverflowcoma5182322809427-10" class="anchor" href="#httpstackoverflowcoma5182322809427-10"></a><a href="http://stackoverflow.com/a/518232/2809427" class="inline-onebox" rel="noopener nofollow ugc">What is the best way to remove accents (normalize) in a Python unicode string? - Stack Overflow</a></h1>
<p>def unicodeToAscii(s):<br>
return ‘’.join(<br>
c for c in unicodedata.normalize(‘NFD’, s)<br>
if unicodedata.category(c) != ‘Mn’<br>
)</p>
<h1><a name="lowercase-trim-and-remove-non-letter-characters-11" class="anchor" href="#lowercase-trim-and-remove-non-letter-characters-11"></a>Lowercase, trim, and remove non-letter characters</h1>
<p>def normalizeString(s):<br>
s = unicodeToAscii(s.lower().strip())<br>
s = re.sub(r"([.!?])“, r” \1", s)<br>
s = re.sub(r"[^a-zA-Z.!?]+“, r” “, s)<br>
s = re.sub(r”\s+“, r” ", s).strip()<br>
return s</p>
<h1><a name="read-queryresponse-pairs-and-return-a-voc-object-12" class="anchor" href="#read-queryresponse-pairs-and-return-a-voc-object-12"></a>Read query/response pairs and return a voc object</h1>
<p>def readVocs(datafile, corpus_name):<br>
print(“Reading lines…”)<br>
# Read the file and split into lines<br>
lines = open(datafile, encoding=‘utf-8’).<br>
read().strip().split(‘\n’)<br>
# Split every line into pairs and normalize<br>
pairs = [[normalizeString(s) for s in l.split(‘\t’)] for l in lines]<br>
voc = Voc(corpus_name)<br>
return voc, pairs</p>
<p>def filterPair(p):<br>
# Input sequences need to preserve the last word for EOS token<br>
return len(p[0].split(’ ‘)) &lt; MAX_LENGTH and len(p[1].split(’ ')) &lt; MAX_LENGTH</p>
<h1><a name="filter-pairs-using-filterpair-condition-13" class="anchor" href="#filter-pairs-using-filterpair-condition-13"></a>Filter pairs using filterPair condition</h1>
<p>def filterPairs(pairs):<br>
return [pair for pair in pairs if filterPair(pair)]</p>
<h1><a name="using-the-functions-defined-above-return-a-populated-voc-object-and-pairs-list-14" class="anchor" href="#using-the-functions-defined-above-return-a-populated-voc-object-and-pairs-list-14"></a>Using the functions defined above, return a populated voc object and pairs list</h1>
<p>def loadPrepareData(corpus, corpus_name, datafile, save_dir):<br>
print(“Start preparing training data …”)<br>
voc, pairs = readVocs(datafile, corpus_name)<br>
print(“Read {!s} sentence pairs”.format(len(pairs)))<br>
pairs = filterPairs(pairs)<br>
print(“Trimmed to {!s} sentence pairs”.format(len(pairs)))<br>
print(“Counting words…”)<br>
for pair in pairs:<br>
voc.addSentence(pair[0])<br>
voc.addSentence(pair[1])<br>
print(“Counted words:”, voc.num_words)<br>
return voc, pairs</p>
<p>save_dir = os.path.join(“data”, “save”)<br>
voc, pairs = loadPrepareData(corpus, corpus_name, datafile, save_dir)</p>
<h1><a name="print-some-pairs-to-validate-15" class="anchor" href="#print-some-pairs-to-validate-15"></a>Print some pairs to validate</h1>
<p>print(“\npairs:”)<br>
for pair in pairs[:10]:<br>
print(pair)</p>
<p>File “”, line 1, in <br>
runfile(‘C:/Users/lab723/Desktop/glove1005/untitled0.py’, wdir=‘C:/Users/lab723/Desktop/glove1005’)</p>
<p>File “C:\Users\lab723\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py”, line 705, in runfile<br>
execfile(filename, namespace)</p>
<p>File “C:\Users\lab723\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py”, line 102, in execfile<br>
exec(compile(f.read(), filename, ‘exec’), namespace)</p>
<p>File “C:/Users/lab723/Desktop/glove1005/untitled0.py”, line 219, in <br>
voc, pairs = loadPrepareData(corpus, corpus_name, datafile, save_dir)</p>
<p>File “C:/Users/lab723/Desktop/glove1005/untitled0.py”, line 209, in loadPrepareData<br>
pairs = filterPairs(pairs)</p>
<p>File “C:/Users/lab723/Desktop/glove1005/untitled0.py”, line 202, in filterPairs<br>
return [pair for pair in pairs if filterPair(pair)]</p>
<p>File “C:/Users/lab723/Desktop/glove1005/untitled0.py”, line 202, in <br>
return [pair for pair in pairs if filterPair(pair)]</p>
<p>File “C:/Users/lab723/Desktop/glove1005/untitled0.py”, line 198, in filterPair<br>
return len(p[0].split(’ ‘)) &lt; MAX_LENGTH and len(p[1].split(’ ')) &lt; MAX_LENGTH</p>
<p>IndexError: list index out of range</p>

Brando_Miranda: <p>I say this with my best interest in mind, but nobody is going to be able to help you with this. Make the question shorter or at least write a more meaningful title…short concise title. No one wants to read a novel. We want to help but your not making it easy for us…</p>